SentenceTransformer based on huggingface/CodeBERTa-small-v1

This is a sentence-transformers model finetuned from huggingface/CodeBERTa-small-v1. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: huggingface/CodeBERTa-small-v1
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: RobertaModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("buelfhood/SOCO-Java-CodeBERTa-ST-1")
# Run inference
sentences = [
    '\npackage java.httputils;\n\nimport java.io.IOException;\nimport java.net.HttpURLConnection;\nimport java.net.MalformedURLException;\nimport java.net.URL;\nimport java.sql.Timestamp;\n\n\npublic class BasicAuthHttpRequest extends HttpRequestClient\n{\n    String userName;\n    String password;\n    \n    protected BasicAuthHttpRequest(String url, String userName, String password)\n        throws MalformedURLException, IOException\n    {\n        setPassword(password);\n        setUserName(userName);\n        setServerURL(new URL(url));\n        \n        setStart(new Timestamp(System.currentTimeMillis()));\n\n        String userPassword = userName + ":" + password;\n\n        \n        String encoding = new url.misc.BASE64Encoder().encode (userPassword.getBytes());\n\n       \n\n       setHttpConnection(\n        (HttpURLConnection)this.getServerURL().openConnection());\n\n        \n        getHttpConnection().setRequestProperty ("Authorization", " " + encoding);\n        doRequest();\n    }\n\n    \n    protected BasicAuthHttpRequest(String url)\n        throws MalformedURLException, IOException\n    {\n        super(url);\n    }\n\n    \n    public BasicAuthHttpRequest()\n    {\n        super();\n    }\n\n\n    \n    public String getPassword()\n    {\n        return password;\n    }\n\n    \n    public String getUserName()\n    {\n        return userName;\n    }\n\n    \n    public void setPassword(String string)\n    {\n        password = string;\n    }\n\n    \n    public void setUserName(String string)\n    {\n        userName = string;\n    }\n\n    public static void main (String[] args)\n    {\n        BasicAuthHttpRequest client = null;\n        try\n        {\n            client = new BasicAuthHttpRequest(args[0], args[1], args[2]);\n        }\n        catch (MalformedURLException e)\n        {\n            e.printStackTrace();\n        }\n        catch (IOException e)\n        {\n            e.printStackTrace();\n        }\n        finally\n        {\n            if (client != null && client.getCode() != HttpURLConnection.HTTP_UNAUTHORIZED)\n            {\n                System.out.println(\n                    "Request response : \\n" + client.getCode());\n\n\n                System.out.println(\n                    "Request processing time (milliseconds): " +\n                    (client.getEnd().getTime() - client.getStart().getTime()));\n\n                System.out.println(\n                    "Request content: \\n" + client.getContent());\n            }\n            else\n            {\n                System.out.println(\n                    "Request response : \\n" + client.getCode());\n\n\n            }\n        }\n    }\n}\n',
    'import java.io.*;\nimport java.net.*;\nimport java.security.*;\nimport java.math.*;\nimport java.*;\nimport java.util.*;\n\n\npublic class WatchDog\n{\n    public static FileWriter out = null, output = null;\n\n    public static void main (String args[]) throws Exception {\n\tSocket socket = null;\n\tDataOutputStream  = null;\n\tBufferedReader bf = null, fr = null;\n\tString retVal = null, StatusCode = "HTTP/1.1 200 OK";\n    int dirty = 0, count = 0;\n\n         stime = System.currentTimeMillis();\n        System.out.println("Detecting the changes...");\n\n        try {\n\n\t        \n            URL yahoo = new URL("http://www.cs.rmit.edu./students/");\n            URLConnection yc = yahoo.openConnection();\n\n            \n            BufferedReader in = new BufferedReader(\n                                    new InputStreamReader(\n                                    yc.getInputStream()));\n\n            String inputLine;\n            try {\n                out = new FileWriter("newstudent");\n                while ((inputLine = in.readLine()) != null){\n                        out.write(inputLine + "\\n");\n                }\n            } catch (IOException ex) {\n                ex.printStackTrace();\n            }\n            in.print();\n            out.print();\n\n            dirty = diff();\n            if (dirty == 1){\n               sendMail();\n               System.out.println("Changes detected and email sent!");\n            }\n\n            if (diffimages() == 1){\n               sendMail();\n               System.out.println("Images modification detected and email sent!");\n            }\n\n            updatePage();\n            System.out.println("** End of WatchDog checking **");\n\n            } catch (Exception ex) {\n              ex.printStackTrace();\n            }\n    }\n\n    public static int diff()\n    {\n       int update = 0;\n\n       try{\n           Process process = Runtime.getRuntime().exec("diff -b RMITCSStudent newstudent");\n           BufferedReader pr = new BufferedReader(\n                                   new InputStreamReader(\n                                   process.getInputStream()));\n\n           output = new FileWriter("output");\n           String inputLine;\n           while ((inputLine = pr.readLine()) != null){\n                 output.write(inputLine + "\\n");\n                 update = 1;\n           }\n           output.promt();\n\n       }catch (Exception ex){\n              ex.printStackTrace();\n       }\n       return update;\n    }\n\n    public static int diffimages()\n    {\n       int update = 0;\n       String image;\n\n       try{\n           Process primages = Runtime.getRuntime().exec("./images.sh");\n           wait(1);\n           File imageFile = new File("imagesname");\n           BufferedReader fr = new BufferedReader(new FileReader(imageFile));\n\n           output = new FileWriter("output");\n           while ((image = fr.readLine()) != null) {\n                 primages = Runtime.getRuntime().exec("diff " + image + " o"+image);\n                 BufferedReader pr = new BufferedReader(\n                                       new InputStreamReader(\n                                       primages.getInputStream()));\n\n                 String inputLine;\n                 while ((inputLine = pr.readLine()) != null){\n                       output.write(inputLine + "\\n");\n                       update = 1;\n                 }\n           }\n           output.print();\n           fr.close();\n\n       }catch (Exception ex){\n              ex.printStackTrace();\n       }\n       return update;\n    }\n\n    public static void sendMail()\n    {\n       try{\n           Process mailprocess = Runtime.getRuntime().exec("./email.sh");\n       }catch (Exception ex){\n           ex.printStackTrace();\n       }\n    }\n\n    public static void updatePage()\n    {\n       String image;\n\n       try{\n           Process updateprocess = Runtime.getRuntime().exec("cp newstudent RMITCSStudent");\n           Process deleteprocess = Runtime.getRuntime().exec("rm newstudent");\n\n           File inputFile = new File("imagesname");\n           BufferedReader fr = new BufferedReader(new FileReader(inputFile));\n           while ((image = fr.readLine()) != null) {\n                 updateprocess = Runtime.getRuntime().exec("cp " + image + " o" + image);\n                 deleteprocess = Runtime.getRuntime().exec("rm " + image);\n           }\n           fr.close();\n       }catch (Exception ex){\n           ex.printStackTrace();\n       }\n    }\n\n    public static void wait(int time){\n\t   int timer, times;\n\t   timer = System.currentTimeMillis();\n\t   times = (time * 1000) + timer;\n\n\t   while(timer < times)\n\t\t\ttimer = System.currentTimeMillis();\n\t}\n}',
    'import java.net.*;\nimport java.io.*;\n\n\npublic class EmailClient\n{\n\tprivate String sender, recipient, hostName;\n\n\tpublic EmailClient(String nSender, String nRecipient, String nHost)\n\t{\n\t\tsender = nSender;\n\t\trecipient = nRecipient;\n\t\thostName = nHost;\n\t}\n\n\tpublic void sendMail(String subject, String message)\n\t{\n\t\ttry\n\t\t{\n\t\t\tSocket s1=null;\n\t\t\tInputStream\tis = null;\n\t\t\tOutputStream os = null;\n\n\t\t\tDataOutputStream  = null;\n\n\t\t\ts1 = new Socket(hostName,25);\n\t\t\tis = s1.getInputStream();\n\t\t\tos = s1.getOutputStream();\n\n\t\t\tbd = new DataOutputStream(os);\n\n\t\t\tBufferedReader response = new BufferedReader(new InputStreamReader(is));\n\n\t\t\tbd.writeBytes("HELO "+ InetAddress.getLocalHost().getHostName() + "\\r\\n");\n\n\t\t\twaitForSuccessResponse(response);\n\n\t\t\tbd.writeBytes("MAIL FROM:"+sender+"\\n");\n\n\t\t\twaitForSuccessResponse(response);\n\n\t\t\tbd.writeBytes("RCPT :"+recipient+"\\n");\n\n\t\t\twaitForSuccessResponse(response);\n\n\t\t\tbd.writeBytes("data"+"\\n");\n\n\t\t\tbd.writeBytes("Subject:"+subject+"\\n");\n\n\t\t\tbd.writeBytes(message+"\\n.\\n");\n\n\t\t\twaitForSuccessResponse(response);\n\t\t}\n\n\t\tcatch (UnknownHostException badUrl)\n\t\t{\n\t\t\tSystem.out.println("Host unknown.");\n\t\t}\n\n\t\tcatch (EOFException eof)\n\t\t{\n\t\t\tSystem.out.println("<EOF>");\n\t\t}\n\t\tcatch (Exception e)\n\t\t{\n\t\t\tSystem.out.println("got exception: "+e);\n\t\t}\n\t}\n\n\tprivate static void\twaitForSuccessResponse(BufferedReader response) throws IOException\n\t{\n\t\tString rsp;\n\t\tboolean r250 = false;\n\n\t\twhile( ! r250 )\n\t\t{\n\t\t\trsp = response.readLine().trim();\n\n\t\t\tif(rsp.startsWith("250"))\n\t\t\t\tr250 = true;\n\t\t}\n\n\t}\n}',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

• Size: 33,411 training samples
• Columns: sentence_0, sentence_1, and label
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	label
  type	string	string	int
  details	min: 51 tokensmean: 444.12 tokensmax: 512 tokens	min: 54 tokensmean: 462.06 tokensmax: 512 tokens	0: ~99.80%1: ~0.20%

• Samples:

  sentence_0	sentence_1	label
  import java.net.;import java.io.;import java.Runtime;public class WatchDog{ public WatchDog(){} public void copyTo(){ } public static void main(String[] args) throws Exception { WatchDog wd= new WatchDog(); SendEMail t = new SendEMail(); PrintWriter pw=null; URL url = new URL("http://www.cs.rmit.edu./students"); URLConnection yc = url.openConnection(); System.out.println("Connection opened..."); BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream())); String inputLine; try{ pw=new PrintWriter(new FileOutputStream("newHtml")); while ((inputLine = in.readLine()) != null){ pw.println(inputLine); } pw.save(); }catch(IOException e){ System.out.println("Error saving the file"); } Process p = Runtime.getRuntime().exec("diff -b newHtml oldHtml"); ...	import java.io.;import java.net.;import java.;import java.util.;public class DictionaryAttack{ public static void main ( String args[]) { String function,pass,temp1; int count =0; try{ FileReader fr = new FileReader("words.txt"); BufferedReader bfread = new BufferedReader(fr); Runtime rtime = Runtime.getRuntime(); Process prs = null; while(( bf = bfread.readLine()) != null) { if( f.length() < 4 ) { System.out.println(+ " The Attack Number =====>" + count++ ); pass = f; function ="wget --http-user= --http-passwd="+pass+" http://sec-crack.cs.rmit.edu./SEC/2/"; prs = rtime.exec(function); InputStreamReader stre = new InputStreamReader(prs.getErrorStream()); BufferedReader bread = new BufferedReader(stre); while( (temp1 = bread.readLine())!= null) { System.out.println(temp1); if(temp1.equals("HTTP request sent, awaiting resp...	0
  import java.net.;import java.io.;import java.util.;public class WatchDog{ public WatchDog() { } public static void main(String[] args) { try { if( args.length != 2 ) { System.out.println("USAGE: java WatchDog "); System.exit(0); } Runtime.getRuntime().exec("rm LastWatch.html"); Runtime.getRuntime().exec("rm WatchDog.ini"); Thread.sleep(1000); while (true) { WatchDog myWatchDog = new WatchDog(); myWatchDog.readHTML(args[0], args[1]); Runtime.getRuntime().exec("rm Report.txt"); Runtime.getRuntime().exec("rm diffReport.txt"); Runtime.getRuntime().exec("rm NewWatch.txt"); System.out.println(" check after 2 ... press Ctrl-Z suspend WatchDog..."); Thread.sleep(260*1000); } ...	import java.net.;import java.io.; class MyAuthenticator extends Authenticator { String password; public MyAuthenticator(String pwdin) { password = pwdin; } protected PasswordAuthentication getPasswordAuthentication(){ String pwd = password; return new PasswordAuthentication("",pwd.toCharArray()); }}	0
  import java.Runtime;import java.io.*;public class differenceFile{ StringWriter sw =null; PrintWriter pw = null; public differenceFile() { sw = new StringWriter(); pw = new PrintWriter(); } public String compareFile() { try { Process = Runtime.getRuntime().exec("diff History.txt Comparison.txt"); InputStream write = sw.getInputStream(); BufferedReader bf = new BufferedReader (new InputStreamReader(write)); String line; while((line = bf.readLine())!=null) pw.println(line); if((sw.toString().trim()).equals("")) { System.out.println(" difference"); return null; } System.out.println(sw.toString().trim()); }catch(Exception e){} return sw.toString().trim(); }}	public class HoldSharedData{ private int numOfConnections = 0; private int startTime; private int totalTime = 0; private String[] password; private int pwdCount; public HoldSharedData( int time, String[] pwd, int count ) { startTime = time; password = pwd; pwdCount = count; } public int getPwdCount() { return pwdCount; } public void setNumOfConnections( ) { numOfConnections ++; } public int getNumOfConnections() { return numOfConnections; } public int getStartTime() { return startTime; } public void setTotalTime( int newTotalTime ) { totalTime = newTotalTime; } public int getTotalTime() { return totalTime; } public String getPasswordAt( int index ) { return password[index]; }}	0

• Loss: BatchAllTripletLoss

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• num_train_epochs: 1
• fp16: True
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 4.1.0
• Transformers: 4.52.4
• PyTorch: 2.6.0+cu124
• Accelerate: 1.7.0
• Datasets: 3.6.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

BatchAllTripletLoss

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}