Skip to content

Commit

Permalink
update prompt eng example in structured output
Browse files Browse the repository at this point in the history
  • Loading branch information
souzatharsis committed Dec 29, 2024
1 parent 35c3aa8 commit 6c8140a
Show file tree
Hide file tree
Showing 8 changed files with 84 additions and 72 deletions.
Binary file modified tamingllms/_build/.doctrees/environment.pickle
View file
Binary file not shown.
Binary file modified tamingllms/_build/.doctrees/notebooks/structured_output.doctree
View file
Binary file not shown.
37 changes: 20 additions & 17 deletions tamingllms/_build/html/_sources/notebooks/structured_output.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -145,7 +145,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -160,11 +160,11 @@
},
{
"cell_type": "code",
"execution_count": 31,
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"prompt = f\"\"\"\n",
"prompt = \"\"\"\n",
"Generate a two-person discussion about the key financial data from the following text in JSON format.\n",
"\n",
"<JSON_FORMAT>\n",
Expand All @@ -179,35 +179,38 @@
" }\n",
"}\n",
"</JSON_FORMAT>\n",
"\n",
"TEXT: {sec_filing}\n",
"\"\"\"\n",
"\n",
"response = client.chat.completions.create(\n",
" model=\"gpt-3.5-turbo\",\n",
" messages=[{\"role\": \"user\", \"content\": prompt}]\n",
" model=\"gpt-4o-mini\",\n",
" messages=[\n",
" {\"role\": \"system\", \"content\": prompt},\n",
" {\"role\": \"user\", \"content\": sec_filing}\n",
" ]\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 32,
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"```json\n",
"{\n",
" \"Person1\": {\n",
" \"name\": \"Alice\",\n",
" \"statement\": \"The revenue for Q1 has increased by 20% compared to last year.\"\n",
" \"statement\": \"The aggregate market value of Apple's stock held by non-affiliates is approximately $2.63 trillion.\"\n",
" },\n",
" \"Person2\": {\n",
" \"name\": \"Bob\",\n",
" \"statement\": \"That's great news! What about the net profit margin?\"\n",
" \"statement\": \"That's impressive! I also noticed that they have around 15.1 billion shares of common stock outstanding.\"\n",
" }\n",
"}\n"
"}\n",
"```\n"
]
}
],
Expand All @@ -218,7 +221,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -234,16 +237,16 @@
},
{
"cell_type": "code",
"execution_count": 33,
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
"False"
]
},
"execution_count": 33,
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
Expand All @@ -256,7 +259,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"We observe the LLM did provide a response in JSON format. However, it is important to note this is not guaranteed. Hence, this strategy may cause failures if dependencies expect output in JSON format."
"We observe the LLM provided a response in JSON format. However, it was enclosed by json markdown tags! While further prompt engineering could lead to a pure JSON response it is important to note this is not guaranteed. Hence, this strategy may cause failures if dependencies expect output in JSON format."
]
},
{
Expand Down
41 changes: 22 additions & 19 deletions tamingllms/_build/html/notebooks/structured_output.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -409,28 +409,29 @@ <h3><a class="toc-backref" href="#id218" role="doc-backlink"><span class="sectio
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">prompt</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;&quot;&quot;</span>
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">prompt</span> <span class="o">=</span> <span class="s2">&quot;&quot;&quot;</span>
<span class="s2">Generate a two-person discussion about the key financial data from the following text in JSON format.</span>

<span class="s2">&lt;JSON_FORMAT&gt;</span>
<span class="si">{</span>
<span class="w"> </span><span class="s2">&quot;Person1&quot;</span><span class="si">:</span><span class="s2"> </span><span class="si">{</span>
<span class="w"> </span><span class="s2">&quot;name&quot;</span><span class="si">:</span><span class="s2"> &quot;Alice&quot;,</span>
<span class="s2"> &quot;statement&quot;: &quot;The revenue for Q1 has increased by 20% compared to last year.&quot;</span>
<span class="s2"> </span><span class="si">}</span><span class="s2">,</span>
<span class="s2"> &quot;Person2&quot;: </span><span class="si">{</span>
<span class="w"> </span><span class="s2">&quot;name&quot;</span><span class="si">:</span><span class="s2"> &quot;Bob&quot;,</span>
<span class="s2">{</span>
<span class="s2"> &quot;Person1&quot;: {</span>
<span class="s2"> &quot;name&quot;: &quot;Alice&quot;,</span>
<span class="s2"> &quot;statement&quot;: &quot;The revenue for Q1 has increased by 20</span><span class="si">% c</span><span class="s2">ompared to last year.&quot;</span>
<span class="s2"> },</span>
<span class="s2"> &quot;Person2&quot;: {</span>
<span class="s2"> &quot;name&quot;: &quot;Bob&quot;,</span>
<span class="s2"> &quot;statement&quot;: &quot;That&#39;s great news! What about the net profit margin?&quot;</span>
<span class="s2"> </span><span class="si">}</span>
<span class="si">}</span>
<span class="s2"> }</span>
<span class="s2">}</span>
<span class="s2">&lt;/JSON_FORMAT&gt;</span>

<span class="s2">TEXT: </span><span class="si">{</span><span class="n">sec_filing</span><span class="si">}</span>
<span class="s2">&quot;&quot;&quot;</span>

<span class="n">response</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">chat</span><span class="o">.</span><span class="n">completions</span><span class="o">.</span><span class="n">create</span><span class="p">(</span>
<span class="n">model</span><span class="o">=</span><span class="s2">&quot;gpt-3.5-turbo&quot;</span><span class="p">,</span>
<span class="n">messages</span><span class="o">=</span><span class="p">[{</span><span class="s2">&quot;role&quot;</span><span class="p">:</span> <span class="s2">&quot;user&quot;</span><span class="p">,</span> <span class="s2">&quot;content&quot;</span><span class="p">:</span> <span class="n">prompt</span><span class="p">}]</span>
<span class="n">model</span><span class="o">=</span><span class="s2">&quot;gpt-4o-mini&quot;</span><span class="p">,</span>
<span class="n">messages</span><span class="o">=</span><span class="p">[</span>
<span class="p">{</span><span class="s2">&quot;role&quot;</span><span class="p">:</span> <span class="s2">&quot;system&quot;</span><span class="p">,</span> <span class="s2">&quot;content&quot;</span><span class="p">:</span> <span class="n">prompt</span><span class="p">},</span>
<span class="p">{</span><span class="s2">&quot;role&quot;</span><span class="p">:</span> <span class="s2">&quot;user&quot;</span><span class="p">,</span> <span class="s2">&quot;content&quot;</span><span class="p">:</span> <span class="n">sec_filing</span><span class="p">}</span>
<span class="p">]</span>
<span class="p">)</span>
</pre></div>
</div>
Expand All @@ -444,16 +445,18 @@ <h3><a class="toc-backref" href="#id218" role="doc-backlink"><span class="sectio
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>{
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>```json
{
&quot;Person1&quot;: {
&quot;name&quot;: &quot;Alice&quot;,
&quot;statement&quot;: &quot;The revenue for Q1 has increased by 20% compared to last year.&quot;
&quot;statement&quot;: &quot;The aggregate market value of Apple&#39;s stock held by non-affiliates is approximately $2.63 trillion.&quot;
},
&quot;Person2&quot;: {
&quot;name&quot;: &quot;Bob&quot;,
&quot;statement&quot;: &quot;That&#39;s great news! What about the net profit margin?&quot;
&quot;statement&quot;: &quot;That&#39;s impressive! I also noticed that they have around 15.1 billion shares of common stock outstanding.&quot;
}
}
```
</pre></div>
</div>
</div>
Expand All @@ -479,12 +482,12 @@ <h3><a class="toc-backref" href="#id218" role="doc-backlink"><span class="sectio
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>True
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>False
</pre></div>
</div>
</div>
</div>
<p>We observe the LLM did provide a response in JSON format. However, it is important to note this is not guaranteed. Hence, this strategy may cause failures if dependencies expect output in JSON format.</p>
<p>We observe the LLM provided a response in JSON format. However, it was enclosed by json markdown tags! While further prompt engineering could lead to a pure JSON response it is important to note this is not guaranteed. Hence, this strategy may cause failures if dependencies expect output in JSON format.</p>
</section>
<section id="json-mode-fine-tuned">
<h3><a class="toc-backref" href="#id219" role="doc-backlink"><span class="section-number">4.3.2. </span>JSON Mode (Fine-Tuned)</a><a class="headerlink" href="#json-mode-fine-tuned" title="Permalink to this heading"></a></h3>
Expand Down
2 changes: 1 addition & 1 deletion tamingllms/_build/html/searchindex.js
View file

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion tamingllms/_build/jupyter_execute/markdown/intro.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
"id": "645b6b42",
"id": "0821d2e9",
"metadata": {},
"source": [
"(intro)=\n",
Expand Down
37 changes: 20 additions & 17 deletions tamingllms/_build/jupyter_execute/notebooks/structured_output.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -145,7 +145,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -160,11 +160,11 @@
},
{
"cell_type": "code",
"execution_count": 31,
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"prompt = f\"\"\"\n",
"prompt = \"\"\"\n",
"Generate a two-person discussion about the key financial data from the following text in JSON format.\n",
"\n",
"<JSON_FORMAT>\n",
Expand All @@ -179,35 +179,38 @@
" }\n",
"}\n",
"</JSON_FORMAT>\n",
"\n",
"TEXT: {sec_filing}\n",
"\"\"\"\n",
"\n",
"response = client.chat.completions.create(\n",
" model=\"gpt-3.5-turbo\",\n",
" messages=[{\"role\": \"user\", \"content\": prompt}]\n",
" model=\"gpt-4o-mini\",\n",
" messages=[\n",
" {\"role\": \"system\", \"content\": prompt},\n",
" {\"role\": \"user\", \"content\": sec_filing}\n",
" ]\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 32,
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"```json\n",
"{\n",
" \"Person1\": {\n",
" \"name\": \"Alice\",\n",
" \"statement\": \"The revenue for Q1 has increased by 20% compared to last year.\"\n",
" \"statement\": \"The aggregate market value of Apple's stock held by non-affiliates is approximately $2.63 trillion.\"\n",
" },\n",
" \"Person2\": {\n",
" \"name\": \"Bob\",\n",
" \"statement\": \"That's great news! What about the net profit margin?\"\n",
" \"statement\": \"That's impressive! I also noticed that they have around 15.1 billion shares of common stock outstanding.\"\n",
" }\n",
"}\n"
"}\n",
"```\n"
]
}
],
Expand All @@ -218,7 +221,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -234,16 +237,16 @@
},
{
"cell_type": "code",
"execution_count": 33,
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
"False"
]
},
"execution_count": 33,
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
Expand All @@ -256,7 +259,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"We observe the LLM did provide a response in JSON format. However, it is important to note this is not guaranteed. Hence, this strategy may cause failures if dependencies expect output in JSON format."
"We observe the LLM provided a response in JSON format. However, it was enclosed by json markdown tags! While further prompt engineering could lead to a pure JSON response it is important to note this is not guaranteed. Hence, this strategy may cause failures if dependencies expect output in JSON format."
]
},
{
Expand Down
Loading

0 comments on commit 6c8140a

Please sign in to comment.