The company plans to release a subscription plan for the tool after it is out of beta.
Adobe’s new AI Assistant instantly generates summaries and insights from long documents, answers questions and formats information for sharing in emails, reports and presentations.
Fusion powers the Sun, and, by extension, makes life on Earth possible.
Researchers use AI to predict and prevent plasma instabilities in fusion reactors, averting reaction disruptions. Experiments show AI forecasts issues 300 milliseconds early, allowing real-time adjustments for stability.
MIT researchers unveil a revolutionary smart glove integrating tactile feedback for enhanced learning, robotics, and virtual reality interactions.
Discover how MIT’s new smart glove is transforming education, robotics, and virtual reality experiences with personalized tactile feedback.
W/ Andrej Karpathy
The Tokenizer is a necessary and pervasive component of Large Language Models (LLMs), where it translates between strings and tokens (text chunks). Tokenizers are a completely separate stage of the LLM pipeline: they have their own training sets, training algorithms (Byte Pair Encoding), and after training implement two fundamental functions: encode() from strings to tokens, and decode() back from tokens to strings. In this lecture we build from scratch the Tokenizer used in the GPT series from OpenAI. In the process, we will see that a lot of weird behaviors and problems of LLMs actually trace back to tokenization. We’ll go through a number of these issues, discuss why tokenization is at fault, and why someone out there ideally finds a way to delete this stage entirely.
Chapters:
00:00:00 intro: Tokenization, GPT-2 paper, tokenization-related issues.
00:05:50 tokenization by example in a Web UI (tiktokenizer)
00:14:56 strings in Python, Unicode code points.
00:18:15 Unicode byte encodings, ASCII, UTF-8, UTF-16, UTF-32
00:22:47 daydreaming: deleting tokenization.
00:23:50 Byte Pair Encoding (BPE) algorithm walkthrough.
00:27:02 starting the implementation.
00:28:35 counting consecutive pairs, finding most common pair.
00:30:36 merging the most common pair.
00:34:58 training the tokenizer: adding the while loop, compression ratio.
00:39:20 tokenizer/LLM diagram: it is a completely separate stage.
00:42:47 decoding tokens to strings.
00:48:21 encoding strings to tokens.
00:57:36 regex patterns to force splits across categories.
01:11:38 tiktoken library intro, differences between GPT-2/GPT-4 regex.
01:14:59 GPT-2 encoder.py released by OpenAI walkthrough.
01:18:26 special tokens, tiktoken handling of, GPT-2/GPT-4 differences.
01:25:28 minbpe exercise time! write your own GPT-4 tokenizer.
01:28:42 sentencepiece library intro, used to train Llama 2 vocabulary.
01:43:27 how to set vocabulary set? revisiting gpt.py transformer.
01:48:11 training new tokens, example of prompt compression.
01:49:58 multimodal [image, video, audio] tokenization with vector quantization.
01:51:41 revisiting and explaining the quirks of LLM tokenization.
02:10:20 final recommendations.
02:12:50??? smile
Exercises:
- Advised flow: reference this document and try to implement the steps before I give away the partial solutions in the video. The full solutions if you’re getting stuck are in the minbpe code https://github.com/karpathy/minbpe/bl…
Links:
Posted in robotics/AI, space
“With the world growing more crowded, the great powers strive to conquer other planets. The race is on. The interplanetary sea has been charted; the first caravelle of space is being constructed. Who will get there first? Who will be the new Columbus?” A robot probe is being readied to explore the secrets of the red planet, Mars. The only component lacking: a human brain. No body. Just the brain. It is needed to deal with unexpected crises in the cold, dark depths of space. The perfect volunteer is found in Colonel Barham, a brilliant but hot-tempered astronaut dying of leukemia. But all goes awry as, stripped of his mortal flesh, Barham — or rather his disembodied brain — is consumed with a newly-found power to control…or destroy. Project psychiatrist Major McKinnon (Grant Williams) diagnoses the brain as having delusions of grandeur…but, just perhaps, Col. Barham has achieved grandeur.
Researchers at Osaka University have developed a groundbreaking flexible optical sensor that works even when crumpled. Using carbon nanotube photodetectors and wireless Bluetooth technology, this sensor enables non-invasive analysis and holds promise for advancements in imaging, wearable technology, and soft robotics. Credit: SciTechDaily.com.
Researchers at Osaka University have created a soft, pliable, and wireless optical sensor using carbon nanotubes and organic transistors on an ultra-thin polymer film. This innovation is poised to open new possibilities in imaging technologies and non-destructive analysis techniques.
Recent years have brought remarkable progress in imaging technology, ranging from high-speed optical sensors capable of capturing more than two million frames per second to compact, lensless cameras that can capture images with just a single pixel.
Reminder: It’s only been 1 YEAR since the Will Smith eating spaghetti videoOpenAI Sora: https://openai.com/sora#researchThumbnail character: BasedAFMKBHD Mer…
Posted in robotics/AI
LoRA Land: 25 fine-tuned #Mistral 7b #LLM that outperform #gpt4 on task-specific applications ranging from sentiment detection to question answering.
Predibase is the fastest way to productionize open-source AI. It enables easy fine-tuning and serving of LLM or deep learning models on cloud infrastructure, promising scalability and cost-effectiveness and is currently used by Fortune 500 and high-growth companies. Try for free: https://predibase.com/free-trial.
Speaker’s Bio: Catherine (Katie) Schuman is a research scientist at Oak Ridge National Laboratory (ORNL). She received her Ph.D. in Computer Science from the University of Tennessee (UT) in 2015, where she completed her dissertation on the use of evolutionary algorithms to train spiking neural networks for neuromorphic systems. She is continuing her study of algorithms for neuromorphic computing at ORNL. Katie has an adjunct faculty appointment with the Department of Electrical Engineering and Computer Science at UT, where she co-leads the TENNLab neuromorphic computing research group. Katie received the U.S. Department of Energy Early Career Award in 2019.
Talk Abstract: Neuromorphic computing is a popular technology for the future of computing. Much of the focus in neuromorphic computing research and development has focused on new architectures, devices, and materials, rather than in the software, algorithms, and applications of these systems. In this talk, I will overview the field of neuromorphic from the computer science perspective. I will give an introduction to spiking neural networks, as well as some of the most common algorithms used in the field. Finally, I will discuss the potential for using neuromorphic systems in real-world applications from scientific data analysis to autonomous vehicles.
Organizer: Dr. Ramtin Zand.
Seminar Webpage: https://www.icaslab.com/Seminar
