AI4Every1 : Chapter 1 (Part 2)

The Rollercoaster Years: Early AI (c. 1950s - 2000)

Alright, so AI had its official launch party at Dartmouth in '56. What happened next? Well, imagine a super-hyped movie launch -- massive expectations, dazzling stars, bold predictions... followed by a mix of genuine hits, some embarrassing flops, and periods where everyone wondered if the whole thing was just smoke and mirrors. The journey from the 1950s to the turn of the millennium was exactly that -- a rollercoaster of soaring optimism and crushing disappointment, punctuated by real, if sometimes limited, progress.

The Golden Age: Anything is Possible! (c. 1950s - early 1970s)

Fresh off the Dartmouth high, the early years were buzzing with excitement. Researchers, armed with newly available computers (still room-sized beasts, mind you) and generous funding (especially from defense agencies like DARPA in the US) [1], truly believed human-level intelligence was just a few years away. This era was dominated by Symbolic AI, often called GOFAI (Good Old-Fashioned AI) [2]. The core idea was that intelligence could be achieved by manipulating symbols according to logical rules, much like how we solve algebra problems or follow a recipe.

We saw some genuinely impressive early programs emerge:

• Logic Theorist (1956): Created by Allen Newell and Herbert Simon (who later won a Nobel in Economics for related work!), this program could prove mathematical theorems from Whitehead and Russell's Principia Mathematica, even finding some proofs considered more elegant than the originals! Talk about showing off early. [3]
• General Problem Solver (GPS): Also by Newell and Simon, this aimed to be, well, general. It tried to solve a variety of simple, formal problems (like logic puzzles) by mimicking human problem-solving strategies (means-ends analysis). Ambitious! [4]
• Samuel's Checkers Player (1959): Arthur Samuel at IBM created a checkers program that could learn from its mistakes and eventually play better than its creator. An early landmark in machine learning! [5]
• SHRDLU (1970): Terry Winograd's program was a star. It operated in a simulated 'blocks world' and could understand and respond to complex natural language commands like "Pick up a big red block and put it on the green cube." It seemed like true language understanding was within reach! [6]

The vibe was electric. Predictions were bold (Marvin Minsky famously predicted in 1970 that "within a generation... the problem of creating 'artificial intelligence' will substantially be solved"). Funding flowed freely. It felt like the dawn of a new intelligent age.

The First AI Winter: Reality Bites (c. mid-1970s - early 1980s)

But then, like hitting unexpected turbulence on a flight, reality set in. The initial optimism crashed against several hard walls: [7]

• The Combinatorial Explosion: Problems that looked simple in principle became computationally impossible as the number of choices grew. Trying all possible chess moves? Forget it. Even simple real-world planning involved too many possibilities for computers of the time.
• The Common Sense Problem: AI programs lacked the vast background knowledge and intuitive understanding of the world that even a child possesses. How do you teach a machine that water is wet, or that you can't be in two places at once? This 'common sense' proved incredibly difficult to encode. (This is related to Moravec's Paradox: things easy for humans, like walking or recognizing faces, are hard for AI, while things hard for humans, like complex calculations, are relatively easy for AI). [8]
• Limited Computing Power: Those room-sized computers were still laughably weak by today's standards.
• The Lighthill Report (UK, 1973) & DARPA Cuts (US): Governments and funding agencies started asking hard questions. Sir James Lighthill's report in the UK was highly critical of AI research, leading to major funding cuts. Similar disillusionment hit the US. The hype bubble burst. AI research funding dried up, labs scaled back, and the field entered its first "AI Winter." It was like the interval after a disappointing first half -- everyone went out for snacks wondering if the second half would be any better. [9]

The Rise of Expert Systems: AI Gets a Job! (c. 1980s)

Just when things looked bleak, AI found a way to make itself useful (and commercially viable). Enter Expert Systems. Instead of trying to replicate general human intelligence, these systems focused on capturing the knowledge of human experts in very specific, narrow domains. Using a set of 'if-then' rules provided by experts, they could offer advice or
make decisions.

• MYCIN: Diagnosed bacterial infections (sometimes more accurately than junior doctors!). [10]
• PROSPECTOR: Helped geologists identify promising sites for mineral deposits. [11]
• XCON: Configured computer systems for DEC (Digital Equipment Corporation), saving them millions. [12]

This was AI finding its niche. Suddenly, businesses were interested again. Funding returned. Japan launched its ambitious Fifth Generation Computer Systems project aiming for AI supremacy. The mantra became 'knowledge is power', and AI seemed to be back in the game, albeit in a more specialized, less grandiose form. [13]

The Second AI Winter: Déjà Vu All Over Again (c. late 1980s - mid 1990s)

Unfortunately, the expert system boom didn't last. The limitations became clear: [14]

• Brittleness: Expert systems worked well within their narrow domain but failed spectacularly if faced with slightly unfamiliar situations. They lacked common sense.
• Knowledge Acquisition Bottleneck: Getting knowledge out of human experts and into rules turned out to be incredibly difficult, slow, and expensive.
• Maintenance Issues: Updating the rule base was a nightmare.
  
• Competition: The specialized hardware(Lisp machines) used for expert systems became obsolete asstandard PCs grew more powerful. The market for expert systems largelycollapsed. Japan's Fifth Generation project also failed to meet itslofty goals. Funding dried up again. Welcome to the Second AI Winter. It felt like a sequel nobody really wanted.

Quiet Progress & Seeds of the Future (c. 1990s)

Despite the gloom, important work continued, often under different names (like machine learning, pattern recognition, or data mining -- AI was still not the coolest term). This period saw the quiet rise of approaches that would dominate the future:

• Machine Learning: Techniques allowing computers to learn from data without being explicitly programmed gained traction. Early neural networks (inspired by brain structure) were being explored, though limited by data and computing power. Probabilistic methods and Bayesian networks offered new ways to handle uncertainty. [15]
• Focus on Specific Problems: Researchers tackled more focused, practical problems like speech recognition, computer vision, and data analysis.
  

  

  Deep Blue in the Computer History Museum and Garry Kasparov. Source: a) James the photographer, CC BY 2.0, Wikimedia, b) Owen Williams, The Kasparov Agency. CC BY-SA 3.0, Wikimedia

• Deep Blue vs. Kasparov (1997): IBM's Deep Blue chess computer defeated world champion Garry Kasparov. While largely based on massive computational power and clever search algorithms (still a form of GOFAI), it was a huge symbolic victory and showed the potential of specialized AI. [16]

This era, though less flashy, laid the crucial groundwork. Computing power was steadily increasing (Moore's Law!) [17], vast amounts of digital data were becoming available (hello, internet!), and new algorithms, especially in machine learning, were maturing. The stage was being set for the AI renaissance of the 21st century. The rollercoaster was heading towards another climb... and this time, it would go much, much higher. $\zeta$

References

1. DARPA's Impact on AI, Progress in AI
2. Wiki, ClapSelf, R//C
3. ResearchGate, Wiki
4. Wiki, Fiveable, Britannica
5. IndiaAI, AS-Wiki, Incomplete Ideas, AIWS
6. MNML, Meta-Guide, Stanford-NLP
7. Wiki, Gabriel-Slides, H20AI
8. Wiki, Latent View
9. ProboAI, UCL, GitHub, Perplexity
10. Wiki, Britannica, SchneppatAI
11. SchneppatAI, Nigel Shadbolt
12. Wiki, Stanford Archive
13. Hist. of AI-Wiki, Wiki, AI Katana, Feigenbaum
14. SmythOS, NewsLetter, DMN, Lisp-Wiki, Reddit, HPE
15. Dataversity, Coursera, ANN-Wiki
16. IBM, EBSCO, Wiki (1, 2)
17. UniteAI, SidecarAI, Nine Altitudes, Iberdrola, ML Timeline Wiki

Take a quick Quiz on things you've read:

The AI Renaissance: Data, GPUs, and Deep Learning (c. 2000 - 2017)

Okay class, buckle up! We're entering the 21st century, and the AI rollercoaster is about to hit a massive incline, powered by a few key ingredients that came together beautifully. Think of it like the perfect storm for intelligent machines, brewing quietly after the second AI winter and then bursting onto the scene. This era wasn't just about gradual improvement; it was about transformative leaps, particularly driven by the resurgence of an old idea supercharged with new
capabilities: Deep Learning.

The Perfect Storm Ingredients:

What changed around the turn of the millennium? Three crucial factors converged:

1. Big Data: The Internet exploded! Suddenly, we weren't just connecting computers; we were generating unimaginable amounts of data -- text, images, videos, clickstreams, and sensor readings. Think of all the photos uploaded to Orkut (remember Orkut?), the emails sent, the websites crawled by Google. This digital deluge, previously seen as a storage problem, became the essential fuel for data-hungry machine learning algorithms. More data meant better training, especially for techniques that learned patterns automatically. [1]
2. Better Algorithms & Renewed Interest in Neural Nets: While the winters were tough, researchers hadn't given up. Techniques like Support Vector Machines (SVMs) showed promise in the late 90s and early 2000s. Crucially, work continued on Artificial Neural Networks (ANNs), inspired by the brain's structure. Early pioneers like Geoffrey Hinton, Yann LeCun, and Yoshua Bengio (often called the 'Godfathers of Deep Learning') persisted, developing better training techniques (like improved backpropagation) and architectures, even when neural nets were out of fashion. [2]
3. Serious Compute Power (Hello, GPUs!): Remember those graphics cards (GPUs) gamers loved for making virtual explosions look awesome? Turns out, their architecture -- designed for massively parallel processing (doing lots of simple calculations simultaneously) -- was perfect for the matrix multiplications at the heart of training deep neural networks. Around the mid-2000s, researchers realized they could repurpose GPUs for general-purpose scientific computing (GPGPU). Suddenly, training complex models that would have taken months became feasible in days or even hours. It was like switching from a bullock cart to a Formula 1 car for computation! [3]

The Deep Learning Breakthrough (The "ImageNet Moment"):

The convergence of these factors led to the 'Big Bang' of modern AI. While progress was steady in areas like spam filtering (using techniques like Naive Bayes) and recommendation systems (like those used by Amazon and Netflix, often using collaborative filtering), the watershed moment came in 2012 at the ImageNet Large Scale Visual Recognition Challenge (ILSVRC).

ImageNet was (and is) a massive database of millions of labelled images. The annual challenge pitted computer vision algorithms against each other: could they correctly identify objects in pictures? For years, progress was incremental. Then, in 2012, a team led by Geoffrey Hinton from the University of Toronto, using a deep convolutional neural network (CNN) called AlexNet (trained on GPUs!), blew the competition away. Their error rate was dramatically lower than any
previous system. It wasn't just a small step; it was a quantum leap.

This "ImageNet Moment" electrified the field. Suddenly, Deep Learning -- using neural networks with many layers ('deep' architectures) trained on large datasets using powerful GPUs -- was demonstrably superior for complex pattern recognition tasks like image classification. Investment poured in, tech giants scrambled to hire deep learning experts, and the techniques rapidly spread to other domains. [4]

Deep Learning Takes Over (c. 2012 - 2017):

The years following AlexNet saw rapid advances fueled by deep learning:

• Computer Vision: Deep CNNs revolutionized image recognition, object detection, and image generation. AI could now 'see' with unprecedented accuracy. Think Facebook automatically tagging your friends (sometimes creepily well), or Google Photos identifying pictures of your dog. [5]
• Speech Recognition: Systems like Google Now, Apple's Siri, and Amazon's Alexa dramatically improved their ability to understand spoken language, thanks largely to deep learning models like Recurrent Neural Networks (RNNs) and later Long Short-Term Memory (LSTM) networks, designed to handle sequential data like speech or text. [6]
• Natural Language Processing (NLP): While still challenging, deep learning started making serious inroads. Techniques like Word Embeddings (e.g., Word2Vec, GloVe) learned to represent words as dense vectors capturing semantic relationships (like 'king' -- 'man' + 'woman' ≈ 'queen'). RNNs and LSTMs were applied to tasks like machine translation and sentiment analysis. [7]
• Reinforcement Learning: Combined with deep learning (Deep Reinforcement Learning or DRL), AI started mastering complex games. Google DeepMind's AlphaGo famously defeated Go world champion Lee Sedol in 2016, a feat previously thought decades away. This wasn't just brute force; AlphaGo learned strategies through self-play. [8]

The Challenge of Sequences and the Road to Attention:

While CNNs excelled at fixed-size inputs like images, and RNNs/LSTMs handled sequences, the latter faced challenges. RNNs processed sequences step-by-step, making it hard to capture long-range dependencies (connecting words far apart in a long sentence) and difficult to parallelize training efficiently. For tasks like machine translation, encoding an entire source sentence into a fixed-size 'context vector' before decoding it into the target language became a bottleneck, especially for long sentences [9]. Researchers developed workarounds, including early forms of 'attention mechanisms' that allowed the decoder to selectively focus on relevant parts of the input sentence while generating each word of the output. This idea -- letting the model 'pay attention' to the most important parts of the input for the current task -- proved incredibly powerful.

This simmering need for better ways to handle sequences, particularly the limitations of RNNs and the promising results of early attention ideas, set the stage for the next major leap. Researchers at Google Brain were working on a novel architecture that would ditch recurrence entirely and rely solely on this attention mechanism. In 2017, they published a paper with a deceptively simple title that would change everything... "Attention Is All You Need." $\eta$

References

1. WEF, PineCone, ImageNet-Wiki, ImageNet-Paper
2. Nvidia, DL-Wiki
3. StackExchange, Wiki, Nvidia-RunAI, Spheron, Pure Storage
4. ResearchGate, The Keyword (Google)
5. History-Comp.Vision, ResearchGate, MDPI-Review, Microsoft, Meta
6. ResearchGate (1, 2), Hinton Paper, PubMed, Google
7. NEU, PubMed (1, 2), IJSSASS, Intro2DL, arXiv (1, 2), BlIT
8. ORMS, UCL, Hacker News, EEVBlog, DeepMind (Google), Paper
9. Seminal Paper, Reddit, MDPI, Zuidema, arXiv, Shelf, Lex, Wiki

The Transformer Era: LLMs, Generative AI, and the Current Boom (2017 - Present)

If the period up to 2017 was the AI Renaissance, then what followed was nothing short of an explosion -- a Cambrian explosion of new architectures, massive models, and capabilities that spilled out of research labs and into our daily lives at breathtaking speed. This current era, the one we're living through right now (yes, you, reading this in April 2025!), is largely defined by one groundbreaking idea and its consequences: the Transformer architecture.

Attention is All You (Really) Need (2017)

Remember how RNNs and LSTMs struggled with processing long sequences step-by-step? It was like trying to understand a long, complex sentence by reading it one word at a time and desperately trying to remember the beginning [1]. In 2017, researchers at Google Brain published a paper titled, with beautiful simplicity, "Attention Is All You Need" [2]. This paper introduced the Transformer. Its revolutionary idea? Ditch the sequential processing of RNNs entirely. Instead, rely solely on a mechanism called self-attention. Think of it like this: when processing a word, the Transformer can 'look' at all other words in the sentence simultaneously, weighing how relevant each word is to the current word, regardless of distance. This allowed models to grasp long-range dependencies much more effectively. Even better, because it wasn't sequential, the architecture was highly parallelizable. You could throw massive amounts of data and compute power (those GPUs again!) at it far more efficiently than ever before. It was a fundamental shift in how machines processed sequential data, especially language.

Enter the Titans: The Rise of Large Language Models (LLMs)

The Transformer architecture became the bedrock for a new generation of language models that scaled to unprecedented sizes:

• BERT (Google, 2018): Bidirectional Encoder Representations from Transformers. BERT was a game-changer because it learned context from both directions (left and right) of a word simultaneously during pre-training (using a clever 'masked language model' objective). This gave it a deeper understanding of language nuances, and it smashed state-of-the-art records across a wide range of NLP tasks like question answering and sentiment analysis. Suddenly, Google Search got smarter, understanding queries better. [3]
• The GPT Dynasty (OpenAI): OpenAI took a different, but equally impactful, approach focusing on generative capabilities: [4]
  • GPT-1 (2018): Demonstrated the power of the Transformer decoder architecture for generative pre-training on diverse internet text. [5]
  • GPT-2 (2019): Scaled up significantly. Its ability to generate remarkably coherent paragraphs of text from a prompt was so impressive (and potentially worrying) that OpenAI initially opted for a staged release due to concerns about misuse. [6]
  • GPT-3 (2020): The true behemoth (175 billion parameters!). GPT-3 didn't just generate text; it showcased stunning few-shot and zero-shot learning. You could often ask it to perform a new task with just a few examples, or sometimes none at all, and it would figure it out. This hinted at more general capabilities emerging from sheer scale. It made LLMs front-page news. [7]
• An LLM Zoo: The success of BERT and GPT spurred intense development. Google followed up with models like LaMDA and PaLM. Meta entered the fray, notably contributing powerful open-source models like Llama (2023) and Llama 2 (2023), democratizing access to capable LLMs. Many others joined the race (e.g., Anthropic's Claude series, Mistral AI).

The Generative AI Explosion (c. 2021 - Present)

The power of these large, pre-trained models, particularly their generative abilities, ignited the Generative AI boom. AI wasn't just understanding data; it was creating it:

• Images from Text: While Generative Adversarial Networks (GANs) had shown promise earlier, Diffusion Models took image generation to stunning new heights around 2021-2022 [8]. Systems like OpenAI's DALL-E 2, Google's Imagen, Stable Diffusion (notably open-source), and Midjourney allowed users to type text prompts ("An astronaut riding a horse in a photorealistic style") and get back incredibly detailed, creative, and often beautiful images. Art, design, and media felt the impact immediately.
• Code Generation: Models trained on vast amounts of code, like OpenAI's Codex (powering GitHub Copilot, launched 2021), began assisting programmers by suggesting code snippets, completing functions, and even writing entire programs from natural language descriptions. Suddenly, coding felt like a collaboration with an AI partner.
• The ChatGPT Moment (Late 2022): OpenAI released ChatGPT, a conversational interface fine-tuned from a GPT-3.5 model (later GPT-4). Its ability to engage in fluent, informative, and versatile conversations, write essays, debug code, brainstorm ideas, and more, captured the public imagination like nothing before. It became the fastest-growing consumer application in history. Google quickly responded with Bard (later rebranded as Gemini), Anthropic launched Claude, and the conversational AI arms race was well and truly on. AI was no longer an abstract concept; it was something millions could interact with directly.

The Bleeding Edge: Where We Are Now (c. 2023 - Early 2025)

The pace hasn't slowed. Key trends shaping the current landscape include:

• Continued Scaling (with Caveats): Models keep getting bigger (e.g., GPT-4 in 2023, Google's Gemini series (2023-2024), Anthropic's Claude 3 (2024), with rumors of even larger models like GPT-5 constantly swirling). These often lead to improved capabilities, but there's growing focus on the immense costs (financial and environmental) and potential diminishing returns, leading to research into more efficient training and architectures.
• Multimodality: The frontier is moving beyond single data types. Models like GPT-4V (2023) and Google's Gemini (2023) can process and reason about combinations of text, images, and sometimes audio, opening up new applications (like describing a picture or analyzing a chart). Video generation (e.g., OpenAI's Sora, 2024) is also rapidly advancing.
• Efficiency and Open Source: Alongside giant proprietary models, there's a huge push for smaller, faster, more efficient models that can run locally or on less powerful hardware. The release of powerful open-source models (like Meta's Llama series - Llama 3 released April 2024, Mistral AI's models) has spurred innovation and wider access, creating a vibrant ecosystem beyond the big tech labs.
• Responsible AI Takes Center Stage: As AI becomes more powerful and pervasive, concerns about ethics, bias, safety, alignment (ensuring AI goals align with human values), job displacement, copyright, and potential misuse are paramount. Research into AI safety, fairness audits, watermarking generated content, and intense global discussions around regulation (like the EU AI Act, finalized 2024) are intensifying. This isn't just a technical challenge; it's a fundamental societal one we're grappling with in real-time.
• AI in India: The wave is definitely being felt here! India has a rapidly growing AI ecosystem with numerous startups, significant investment from tech giants setting up research hubs, dedicated AI programs in academia (like several IITs and IIITs), and government initiatives like the IndiaAI mission (approved March 2024) aiming to build compute infrastructure and foster development and adoption across sectors like healthcare, agriculture, and education. We're not just users; we're increasingly becoming creators and innovators in the global AI landscape, contributing unique perspectives and solutions.

So, here we are in April 2025. AI development is faster, more impactful, and more widely discussed than ever before. From ancient myths about automatons to algorithms that can generate photorealistic images, write complex code, hold nuanced conversations, and even discover new scientific insights, it's been an incredibly wild ride. And the most exciting (or perhaps daunting) part? This story is clearly still being written, chapter by chapter, and almost day by day. What comes next... well, that's what the rest of this book is about! $\theta$

References

1. Stack Exchange, AppInventiv
2. Wiki, Paper, Labellerr, Reddit
3. Paper, Kaggle, Synced, ACL Anthology, Brave River, Towards Data Sc.
4. Medium, HGS, Wiki
5. Medium, Wiki, Kaggle, TDS
6. Paper, Alammar, GitHub, Wiki, Hugging Face (Docs, Community), Blog
7. Wiki, Open AI, Paper-Code, Twilio, Springboard
8. Intro, Medium, Coursera, Analytics Vidhya, Paper, Kanerika, Pytorch Demo

Key LLM Milestones (2017-2023)

Year	Model Name	Originator(s)	Key Contribution / Significance	Parameters (Approx.)
2017	Transformer	Google et al.	Foundational architecture (Attention is All You Need paper)	N/A (Architecture)
2018	BERT	Google	Deep Bidirectional understanding (MLM), SOTA on NLU tasks	Up to 340M
2018	GPT-1	OpenAI	Generative Pre-training with Transformer Decoder	117M
2019	GPT-2	OpenAI	Scaled generation, coherent text, staged release due to concerns	Up to 1.5B
2020	GPT-3	OpenAI	Massive scale, few/zero-shot learning, public impact	175B
2021	LaMDA	Google	Optimized for dialogue applications	137B
2022	PaLM	Google	Scaled to 540B, advanced reasoning	540B
2023	Llama	Meta	High-performance model released to researchers	Up to 65B
2023	Llama 2	Meta	Open-sourced for research & commercial use	Up to 70B
2023+	Claude Series	Anthropic	Focus on AI safety, conversational ability	Various
2023+	Mistral Models	Mistral AI	High-performance open-source models (incl. MoE)	Various

PDF of this Chapter: