DoctorGPT (A Decoder-Only Implementation of GPT2 Architecture)

Jeethu Srinivas A; Hari Prechetha K; Kavin Devraj

DoctorGPT (A Decoder-Only Implementation of GPT2 Architecture)

November 2023

Vol-9, Issue-6

Paper ID: 22101

ISSN: 2395-4396

Downloads: 0

Abstract & Details

Research Area

Artificial Intelligence

Keywords

Natural Language Processing (NLP) Decoder-Only Transformer Text Generation Deep Learning PyTorch Autoregressive Generation Self-Attention Mechanism Benchmark Datasets Model Evaluation Long-Range Dependencies BLEU Score Perplexity Machine Translation Text Summarization Creative Writing

Abstract

Natural language processing (NLP) has developed dramatically in recent years, particularly with the introduction of transformer-based architectures in deep learning. Among these, decoder-only transformers stand out as a straightforward yet promising approach to text generation. This project focuses on developing and testing a decoder-only transformer model for text creation jobs. A crucial part of NLP is text creation, which includes synthesizing human-like text from provided inputs. Traditional techniques, like as n-gram models, have limitations in terms of understanding long-distance relationships and producing cohesive text. However, decoder-only transformers that use self-attention and autoregressive generation have demonstrated superior text creation capabilities. The key goals here are to build a decoder-only transformer model with PyTorch, train it on large amounts of text input, and evaluate its ability to generate high-quality text. Data cleansing, creating the model's architecture, training with optimization approaches, and evaluating performance using measures such as perplexity and BLEU score are all part of the process. The results show the model's ability to generate coherent and contextually relevant content, demonstrating its promise in machine translation, text summarization, and creative writing.

License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Author Information

#	Name	Institute / Affiliation
1	Jeethu Srinivas A	Bannari Amman Institute of Technology
2	Hari Prechetha K	Bannari Amman Institute of Technology
3	Kavin Devraj	Bannari Amman Institute of Technology

How to Cite

Use the following formats to cite this article in your research.

APA Style

A, Jeethu Srinivas, K, Hari Prechetha, & Devraj, Kavin (2023). DoctorGPT (A Decoder-Only Implementation of GPT2 Architecture). International Journal of Advance Research and Innovative Ideas In Education, 9(6), 1002-1008.

MLA Style

A, Jeethu Srinivas, et al. "DoctorGPT (A Decoder-Only Implementation of GPT2 Architecture)." International Journal of Advance Research and Innovative Ideas In Education, vol. 9, no. 6, 2023, pp. 1002-1008.

IEEE Style

Jeethu Srinivas A, Hari Prechetha K, and Kavin Devraj, "DoctorGPT (A Decoder-Only Implementation of GPT2 Architecture)," International Journal of Advance Research and Innovative Ideas In Education, vol. 9, no. 6, pp. 1002-1008, 2023.

Vancouver Style

A Jeethu Srinivas, K Hari Prechetha, Devraj Kavin. DoctorGPT (A Decoder-Only Implementation of GPT2 Architecture). International Journal of Advance Research and Innovative Ideas In Education. 2023;9(6):1002-1008.

Harvard Style

A, Jeethu Srinivas, K, Hari Prechetha, & Devraj, Kavin (2023) 'DoctorGPT (A Decoder-Only Implementation of GPT2 Architecture)', International Journal of Advance Research and Innovative Ideas In Education, 9(6), pp. 1002-1008.

Chicago Style

A, Jeethu Srinivas, Hari Prechetha K, and Kavin Devraj. "DoctorGPT (A Decoder-Only Implementation of GPT2 Architecture)." International Journal of Advance Research and Innovative Ideas In Education 9, no. 6 (2023): 1002-1008.

Turabian Style

A, Jeethu Srinivas, Hari Prechetha K, and Kavin Devraj. "DoctorGPT (A Decoder-Only Implementation of GPT2 Architecture)." International Journal of Advance Research and Innovative Ideas In Education 9, no. 6 (2023): 1002-1008.