I haven’t had much exposure to LLMs and wanted to experiment with them a bit. I figured a good first project would be to try and generate summaries of these posts as an extra section. I had to do a little bit of research but in the beginning but after some false starts, the whole process was surprisingly easy and straightforward.
Ollama and Llama 3.2
The first step is to install Ollama. You can follow the instructions on the site but the command is basically
curl -fsSL https://ollama.com/install.sh | sh
Ollama is a tool that allows you to download LLMs from a repository and run it
locally on your machine. I looked into
llama-cpp as well but using that
seemed a bit more involved that what I needed. Ollama seems to be sort of a
wrapper around llama-cpp.
The install script didn’t start ollama for me automatically so I had to run
ollama serve manually. To download a model just do. After installing and
starting Ollama, you can download LLM models from their repository. I didn’t
really need anything too big or powerful so I just decided on llama3.2.
ollama pull llama3.2
I ran into a minor issue where I couldn’t download any models and kept getting a
401 error. This issue was that my WSL time wasn’t synchronized. To fix that I
installed ntpdate and ran ntpdate us.pool.ntp.org.
After downloading a model, you can run it and give it prompts like
ollama run llama3.2 "Summarize this post: $(cat test.md)"
Generate summaries of Hugo posts with Llama 3.2
Once you have ollama running then you’re pretty much done. I created a simple bash script to generate summaries of a file. It’s pretty hacky but does the job.
#!/bin/bash
post=$1
echo "" >> $post
echo "## TLDR (auto-generated with emojis)" >> $post
ollama run llama3.2 "Summarize this post using only emojis: $(cat $post)" | fold -s >> $post
ollama run llama3.2 "Generate a short summary of this post: $(cat $post)" | fold -s >> $post
I can run the script with ./summarize.sh post.md. Running the script on my
laptop took around 70 seconds. I tested out llama3.2:1b as well and that took
around 33 seconds so about half the time.
Testing
For fun, I ran a modified test script to auto generate summaries for the two different models.
#!/bin/bash
post=$1
## Queries
emoji_summary_query="Summarize this post using only emojis:"
short_summary_query="Give a short summary of this post:"
markdown_summary_query="Give a short summary of this post in markdown:"
## Models to test
models=("llama3.2" "llama3.2:1b")
echo ""
for model in "${models[@]}"; do
echo "## TLDR (auto-generated with $model)"
echo "Query: $emoji_summary_query"
echo ""
ollama run $model "$emoji_summary_query $(cat $post)"
echo "Query: $short_summary_query"
echo ""
ollama run $model "$short_summary_query $(cat $post)" | fold -s
echo "### Summary in markdown (auto-generated with $model)"
echo "Query: $markdown_summary_query"
echo ""
ollama run $model "$markdown_summary_query $(cat $post)" | fold -s
done
The unedited output of the script should be below.
TLDR (auto-generated with llama3.2)
Query: Summarize this post using only emojis:
📚💻🔍💡🎉👍
Query: Give a short summary of this post:
This post is about experimenting with Large Language Models (LLMs) using the Ollama tool, specifically with the Llama 3.2 model. The author installs Ollama and downloads the Llama 3.2 model, then creates a simple bash script to generate summaries of Hugo posts using the model. They test the script with two different models, comparing their performance times. The post provides step-by-step instructions on how to install Ollama and use it to generate summaries.
Summary in markdown (auto-generated with llama3.2)
Query: Give a short summary of this post in markdown:
Experiment With LLMs
Overview
This post is about experimenting with Large Language Models (LLMs) using Ollama and Llama 3.2.
Installation and Setup
To install Ollama, run the following command:
curl -fsSL https://ollama.com/install.sh | sh
After installation, download a model by running:
ollama pull llama3.2
Run Ollama with the serve command to start it.
Generating Summaries
Create a simple bash script to generate summaries of a file using Llama 3.2. The script takes a post as input and appends auto-generated summaries to the original post.
#!/bin/bash
post=$1
echo "" >> $post
echo "## TLDR (auto-generated with emojis)" >> $post
ollama run llama3.2 "Summarize this post using only emojis: $(cat $post)" |
fold -s >> $post
ollama run llama3.2 "Generate a short summary of this post: $(cat $post)" |
fold -s >> $post
Run the script with ./summarize.sh post.md to generate summaries.
Testing
Create a test script to auto-generate summaries for two different models:
#!/bin/bash
post=$1
## Queries
emoji_summary_query="Summarize this post using only emojis:"
short_summary_query="Give a short summary of this post:"
markdown_summary_query="Give a short summary of this post in markdown:"
## Models to test
models=("llama3.2" "llama3.2:1b")
echo ""
for model in "${models[@]}"; do
echo "## TLDR (auto-generated with $model)"
echo "Query: $emoji_summary_query"
time ollama run $model "$emoji_summary_query $(cat $post)" | fold -s
echo "Query: $short_summary_query"
time ollama run $model "$short_summary_query $(cat $post)" | fold -s
echo "### Summary in markdown"
echo "Query: $markdown_summary_query"
time ollama run $model "$markdown_summary_query $(cat $post)" | fold -s
done
Run the test script with ./test.sh post.md to generate summaries for two
models.
TLDR (auto-generated with llama3.2:1b)
Query: Summarize this post using only emojis:
🎯💻📝+++ 🏠🔧💸 Ollama and Llama 3.2 installation 💡 👍 Easy to set up and use 🙌 😐 Minor issues with model downloading 🔴 🕒️ Generate summaries in around 70-100 seconds ⏱️ 🎉 Testing and experimenting with different models 📊
Llama 3.2 Installation
🔧 Download Ollama using script: curl -fsSL https://ollama.com/install.sh | sh
💻 Install Llama 3.2 manually by running ollama serve and then downloading the model
Generating Summaries with Llama 3.2
📝 Create a bash script to generate summaries of Hugo posts using Ollama: #!/bin/bash post=$1 echo "" >> $post echo "## TLDR (auto-generated with emojis)" >> $post ollama run llama3.2 "Summarize this post: $(cat test.md)" | fold -s >> $post
🚀 Run the script to generate summaries of a single Hugo post
Testing Llama 3.2 Models
📊 Create a modified test script to auto-generate summaries for different models, including:
🔍 Llama 3.2
⏱️ Llama 3.2:1b
📝 Run the scripts with ./summarize.sh post.md
Query: Give a short summary of this post:
This post discusses experimenting with Large Language Models (LLMs) like LLaMA 3.2 and OLLAMA, a tool for downloading and running LLMs locally on a machine. The author reviews the installation process and uses OLLAMA to download a model and then generates summaries of Hugo posts using it.
The author also explores generating summaries with Llama 3.2 by creating a bash script that downloads a model from a repository, runs it, and includes it in a file as markdown or an auto-generated summary with emojis.
Additionally, the author tests OLLAMA and Llama 3.2 by running modified test scripts to generate summaries for both models. The test script uses a series of prompts to simulate user queries on these models.
Summary in markdown (auto-generated with llama3.2:1b)
Query: Give a short summary of this post in markdown:
Experiment With LLMs
This post provides an introduction to using Large Language Models (LLMs) for text generation and summarization.
Installing Ollama
The first step is to install Ollama, a tool that allows
downloading LLMs from a repository and running them locally. The installation
script uses curl to download the model, which can be customized as needed.
Downloading Llama 3.2 Models
The author chose llama3.2 for their initial experiment, but found it too
complex. To fix this, they installed ntpdate and used it to synchronize their
local time with a remote server.
Generating Summaries using Ollama
After setting up Ollama, the author created a simple bash script to generate summaries of Hugo posts using Llama 3.2. The script uses the model’s output as input for additional summary tasks.
Testing the Script
The author also tested their script by running it on two different models:
llama3.2 and llama3.2:1b. Results showed that both models were able to
generate summaries in a reasonable amount of time.
Conclusion
This experiment demonstrates how easy it is to use LLMs for text generation and summarization, even for those with limited experience. The author’s script provides a basic framework for getting started, and can be customized as needed to suit specific use cases.
TL;DR (auto-generated with llama3.2:1b)
Here is the summary in markdown format:
Experiment With LLMs
========================
This post provides an introduction to using Large Language Models (LLMs) for text generation and summarization.
Installing Ollama
The first step is to install Ollama, a tool that allows
downloading LLMs from a repository and running them locally. The installation
script uses curl to download the model, which can be customized as needed.
Downloading Llama 3.2 Models
The author chose llama3.2 for their initial experiment, but found it too
complex. To fix this, they installed ntpdate and used it to synchronize their
local time with a remote server.
Generating Summaries using Ollama
After setting up Ollama, the author created a simple bash script to generate summaries of Hugo posts using Llama 3.2. The script uses the model’s output as input for additional summary tasks.
Testing the Script
The author also tested their script by running it on two different models:
llama3.2 and llama3.2:1b. Results showed that both models were able to
generate summaries in a reasonable amount of time.
Summary in markdown (auto-generated with llama3.2:1b)
Query: Give a short summary of this post in markdown:
Experiment With LLMs
This experiment demonstrates how easy it is to use LLMs for text generation and summarization, even for those with limited experience. The author’s script provides a basic framework for getting started, and can be customized as needed to suit specific use cases.
TL;DR (auto-generated with llama3.2:1b)
Here is the modified test script to auto-generate summaries for different models:
import subprocess
import json
import argparse
def summarize_post(post, model):
## Download the model from a repository using curl
curl_url = f"https://github.com/{model['repo']}/archive/master.zip"
download_command = f"curl -o {model['output_file']} {curl_url}"
## Run the model and generate markdown output
run_command = f"{model['script']} --input={post.content}
--output={model['output_file']}\n"
subprocess.run(run_command, shell=True)
## Load the generated summary into a JSON object
with open(model['output_file'], 'r') as file:
summary_json = json.load(file)
## Return the markdown format string for the summary
return f"**Summary:**\n\n{summary_json}"
def auto_generate_summaries(models, post):
summaries = []
for model in models:
if model['repo'] == 'llama3.2':
summaries.append(summarize_post(post, model))
elif model['repo'] == 'ollama':
summaries.append("No summary generated for OLLAMA")
return summaries
def main():
parser = argparse.ArgumentParser(description="Generate summaries from LLMs")
models_group = parser.add_mutually_exclusive_group()
models_group.add_argument("--llama3.2", action='store_true')
models_group.add_argument("--ollama", action='store_true')
post_file = "post.md"
output_dir = "summaries"
args = parser.parse_args()
summaries = auto_generate_summaries(models, {"name": "post", "content":
open(post_file).read()})
## Add the downloaded models to the summary
for model in models:
if model['repo'] == 'llama3.2':
summaries.append(f"Downloaded Llama 3.2: {model['output_file']} and
added it to the summary")
elif model['repo'] == 'ollama':
summaries.append("No downloaded models found for OLLAMA")
## Save the summaries to a directory
for i, summary in enumerate(summaries):
output_file = f"{output_dir}/{i+1}.md"
with open(output_file, "w") as file:
file.write(summary)
## Run the test script on the first model and save its results to the
summary
if i == 0:
print(f"Running Llama 3.2: {model['output_file']}\n")
subprocess.run(["./summarize.sh", post_file])
print("All summaries generated successfully!")
if __name__ == "__main__":
main()
This script uses the subprocess module to download models and run scripts,
and the argparse module to parse command-line arguments. It also uses a
custom function auto_generate_summaries to generate the summaries for
different models.
Note that this script assumes that you have already downloaded the LLMs from
their respective repositories using curl. You will need to modify the script
to download the LLMs if they are not already available in your repository.