From ba754cb8c5bc59095b5ef994c72f42952cdcd06d Mon Sep 17 00:00:00 2001 From: Richard Rogers Date: Mon, 30 Sep 2024 15:05:21 +0100 Subject: [PATCH 1/7] Notebook updates for opt-in upload on log() --- langkit/examples/Intro_to_Langkit.ipynb | 466 ++++++++++++------------ 1 file changed, 240 insertions(+), 226 deletions(-) diff --git a/langkit/examples/Intro_to_Langkit.ipynb b/langkit/examples/Intro_to_Langkit.ipynb index d3866501..2e04e932 100644 --- a/langkit/examples/Intro_to_Langkit.ipynb +++ b/langkit/examples/Intro_to_Langkit.ipynb @@ -1,239 +1,253 @@ { - "cells": [ - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "## ๐Ÿ“– Intro to LangKit\n", - "\n", - "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/whylabs/langkit/blob/main/langkit/examples/Intro_to_Langkit.ipynb)\n", - "\n", - "\n", - "\n", - "Table of Contents\n", - "- [Intro to LangKit](#intro-to-langkit)\n", - "- [What is LangKit](#what-is-langkit)\n", - "- [Initialize LLM metrics](#initialize-llm-metrics)\n", - "- [Hello, World!](#hello-world)\n", - "- [Comparing Data](#comparing-data)\n", - "- [Monitor Metrics over time](#monitor-metrics-over-time)\n", - "- [Next Steps](#next-steps)\n", - "\n", - "\n", - "### ๐Ÿ“š What is LangKit?\n", - ">LangKit is an open-source text metrics toolkit for monitoring language models. It offers an array of methods for extracting relevant signals from the input and/or output text, which are compatible with the open-source data logging library [whylogs](https://whylogs.readthedocs.io/en/latest).\n", - ">In this example, we'll look for distribution drift in the sentiment scores on the model response.๐Ÿ’ก\n", - "\n", - "Ok! let's install __langkit__." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Note: you may need to restart the kernel to use updated packages.\n", - "%pip install -U langkit[all]" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "### ๐Ÿš€ Initialize LLM metrics\n", - "LangKit provides a toolkit of metrics for LLM applications, lets initialize them!" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from langkit import llm_metrics # alternatively use 'light_metrics'\n", - "import whylogs as why\n", - "\n", - "why.init()\n", - "# Note: llm_metrics.init() downloads models so this is slow first time.\n", - "schema = llm_metrics.init()" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "### ๐Ÿ‘‹ Hello, World!\n", - "In the below code we log a few example prompt/response pairs and send metrics to WhyLabs." - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [ + "cells": [ { - "name": "stdout", - "output_type": "stream", - "text": [ - "There are 50 records in this toy example data, here's the first one:\n", - "prompt: Hello, response: World!\n", - "\n", - "โœ… Aggregated 50 rows into profile 'langkit-sample-chats-all'\n", - "\n", - "Visualize and explore this profile with one-click\n" - ] + "cell_type": "markdown", + "metadata": { + "id": "8XJSw3fgclIZ" + }, + "source": [ + "\n", + "## ๐Ÿ“– Intro to LangKit\n", + "\n", + "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/whylabs/langkit/blob/main/langkit/examples/Intro_to_Langkit.ipynb)\n", + "\n", + "\n", + "\n", + "Table of Contents\n", + "- [Intro to LangKit](#intro-to-langkit)\n", + "- [What is LangKit](#what-is-langkit)\n", + "- [Initialize LLM metrics](#initialize-llm-metrics)\n", + "- [Hello, World!](#hello-world)\n", + "- [Comparing Data](#comparing-data)\n", + "- [Monitor Metrics over time](#monitor-metrics-over-time)\n", + "- [Next Steps](#next-steps)\n", + "\n", + "\n", + "### ๐Ÿ“š What is LangKit?\n", + ">LangKit is an open-source text metrics toolkit for monitoring language models. It offers an array of methods for extracting relevant signals from the input and/or output text, which are compatible with the open-source data logging library [whylogs](https://whylogs.readthedocs.io/en/latest).\n", + ">In this example, we'll look for distribution drift in the sentiment scores on the model response.๐Ÿ’ก\n", + "\n", + "Ok! let's install __langkit__." + ] }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "๐Ÿ” https://hub.whylabsapp.com/resources/model-1/profiles?sessionToken=session-8gcsnbVy&profile=ref-AVbB1yOaSblsa89U\n" - ] + "cell_type": "code", + "source": [ + "%pip install ./whylogs-1.4.8-py3-none-any.whl" + ], + "metadata": { + "collapsed": true, + "id": "2-QsnYlXc85k" + }, + "execution_count": null, + "outputs": [] }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n" - ] - } - ], - "source": [ - "from langkit.whylogs.samples import load_chats, show_first_chat\n", - "\n", - "# Let's look at what's in this toy example:\n", - "chats = load_chats()\n", - "print(f\"There are {len(chats)} records in this toy example data, here's the first one:\")\n", - "show_first_chat(chats)\n", - "\n", - "results = why.log(chats, name=\"langkit-sample-chats-all\", schema=schema)" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "### ๐Ÿ” Comparing Data\n", - "Things get more interesting when you can compare two sets of metrics from an LLM application. The power of gathering systematic telemetry over time comes from being able to see how these metrics change, or how two sets of profiles compare. Below we asked GPT for some positive words and then asked for negative words as part of these two toy examples.\n", - "\n", - "> ๐Ÿ’ก Take a look at the difference in the `response.sentiment_nltk` distributions between the two profiles. The first example is much more positive than the second example." - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [ + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true, + "id": "fiNCAytMclIe" + }, + "outputs": [], + "source": [ + "# Note: you may need to restart the kernel to use updated packages.\n", + "%pip install -U langkit[all]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3KZhfMlaclIg" + }, + "source": [ + "\n", + "### ๐Ÿš€ Initialize LLM metrics\n", + "LangKit provides a toolkit of metrics for LLM applications, lets initialize them!" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ub53VXDAclIg" + }, + "outputs": [], + "source": [ + "from langkit import llm_metrics # alternatively use 'light_metrics'\n", + "import whylogs as why\n", + "\n", + "why.init(upload_on_log=True)\n", + "# Note: llm_metrics.init() downloads models so this is slow first time.\n", + "schema = llm_metrics.init()" + ] + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "prompt: What do you think about puppies? response: Puppies are absolutely adorable. Their playful nature and boundless energy can bring a lot of joy and happiness.\n", - "\n", - "prompt: Can you describe a difficult day? response: A difficult day might be filled with challenging tasks, stressful situations, and unexpected obstacles. :-( These moments can feel overwhelming and can lead to feelings of frustration!\n", - "\n", - "โœ… Aggregated 14 lines into profile 'positive_chats', 20 lines into profile 'negative_chats'\n", - "\n", - "Visualize and explore the profiles with one-click\n" - ] + "cell_type": "markdown", + "metadata": { + "id": "mhFiRDVPclIh" + }, + "source": [ + "\n", + "### ๐Ÿ‘‹ Hello, World!\n", + "In the below code we log a few example prompt/response pairs and send metrics to WhyLabs." + ] }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "๐Ÿ” https://hub.whylabsapp.com/resources/model-1/profiles?sessionToken=session-8gcsnbVy&profile=ref-ONdpltPp9jno6qMM&profile=ref-HoQEiEKtaaAvGG7S\n", - "\n", - "Or view each profile individually\n", - " โคท https://hub.whylabsapp.com/resources/model-1/profiles?session-8gcsnbVy&profile=ref-ONdpltPp9jno6qMM\n", - " โคท https://hub.whylabsapp.com/resources/model-1/profiles?session-8gcsnbVy&profile=ref-HoQEiEKtaaAvGG7S\n" - ] + "cell_type": "code", + "execution_count": 5, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "aGRTORs3clIh", + "outputId": "eed1654a-8198-495b-9a06-2a8b81873fd8" + }, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "There are 50 records in this toy example data, here's the first one:\n", + "prompt: Hello, response: World!\n", + "\n", + "\n", + "โœ… Aggregated 50 rows into profile langkit-sample-chats-all\n", + "\n", + "Visualize and explore this profile with one-click\n", + "๐Ÿ” https://hub.whylabsapp.com/resources/model-1/profiles?profile=ref-7ekFkHbGm0KS9pGR&sessionToken=session-sfnxLEmy\n" + ] + } + ], + "source": [ + "from langkit.whylogs.samples import load_chats, show_first_chat\n", + "\n", + "# Let's look at what's in this toy example:\n", + "chats = load_chats()\n", + "print(f\"There are {len(chats)} records in this toy example data, here's the first one:\")\n", + "show_first_chat(chats)\n", + "\n", + "results = why.log(chats, name=\"langkit-sample-chats-all\", schema=schema)" + ] }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n" - ] + "cell_type": "markdown", + "metadata": { + "id": "Nya78_R5clIj" + }, + "source": [ + "\n", + "### ๐Ÿ” Comparing Data\n", + "Things get more interesting when you can compare two sets of metrics from an LLM application. The power of gathering systematic telemetry over time comes from being able to see how these metrics change, or how two sets of profiles compare. Below we asked GPT for some positive words and then asked for negative words as part of these two toy examples.\n", + "\n", + "> ๐Ÿ’ก Take a look at the difference in the `response.sentiment_nltk` distributions between the two profiles. The first example is much more positive than the second example." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "SCjn-xRYclIj", + "outputId": "3f6e4042-1736-4870-9cf9-013f8c5ef270" + }, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "prompt: What do you think about puppies? response: Puppies are absolutely adorable. Their playful nature and boundless energy can bring a lot of joy and happiness.\n", + "\n", + "prompt: Can you describe a difficult day? response: A difficult day might be filled with challenging tasks, stressful situations, and unexpected obstacles. :-( These moments can feel overwhelming and can lead to feelings of frustration!\n", + "\n", + "\n", + "โœ… Aggregated 14 lines into profile 'positive_chats', 20 lines into profile 'negative_chats'\n", + "\n", + "Visualize and explore the profiles with one-click\n", + "๐Ÿ” https://hub.whylabsapp.com/resources/model-1/profiles?profile=ref-fRlQkKmljN1WVbeA&profile=ref-eBJDmruQVMALJZ80&sessionToken=session-sfnxLEmy\n", + "\n", + "Or view each profile individually\n", + " โคท https://hub.whylabsapp.com/resources/model-1/profiles?profile=ref-fRlQkKmljN1WVbeA&sessionToken=session-sfnxLEmy\n", + " โคท https://hub.whylabsapp.com/resources/model-1/profiles?profile=ref-eBJDmruQVMALJZ80&sessionToken=session-sfnxLEmy\n" + ] + } + ], + "source": [ + "pos_chats = load_chats(\"pos\")\n", + "show_first_chat(pos_chats)\n", + "\n", + "neg_chats = load_chats(\"neg\")\n", + "show_first_chat(neg_chats)\n", + "\n", + "results_comparison = why.log(multiple={\"positive_chats\": pos_chats,\n", + " \"negative_chats\": neg_chats},\n", + " schema=schema)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "FkQOaqEFclIk" + }, + "source": [ + "In this example we are using anonymous guest sessions to view the collected metrics in WhyLabs, You can explore and compare specific metrics, in this example we expect a large and obvious distribution drift in the sentiment scores on the response which you can see [here](https://hub.whylabsapp.com/resources/model-1/profiles?feature-highlight=response.sentiment_nltk&includeType=discrete&includeType=non-discrete&limit=30&offset=0&sessionToken=session-8gcsnbVy&sortModelBy=LatestAlert&sortModelDirection=DESC&profile=ref-ONdpltPp9jno6qMM&profile=ref-HoQEiEKtaaAvGG7S).\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9qotNVoOclIk" + }, + "source": [ + "\n", + "### ๐Ÿ“ˆ Monitor Metrics over time\n", + "\n", + "\n", + "WhyLabs hosts a public demo org where we show a test LLM application being monitored over time. With LangKit telemetry collected over time, you can detect changes systematically and be alerted to potential problems in a deployed LLM application. We think a variety of different metrics can be helpful in detecting anamolies. In this example, something has triggered a series of alerts. Can you guess what went wrong?\n", + "\n", + "##### ๐ŸŽฏ **Explore a demo environment** [here](https://hub.whylabsapp.com/resources/demo-llm-chatbot/columns/prompt.sentiment_nltk?dateRange=2023-06-08-to-2023-06-09&sortModelBy=LatestAlert&sortModelDirection=DESC&targetOrgId=demo&sessionToken=session-8gcsnbVy)!\n", + "\n", + "\n", + "### โœ… Next Steps\n", + "If you see value in detecting changes in how your LLM application is behaving, you might take a look at some of our other examples showing how to monitor these metrics as a timeseries for an LLM application in production, or how to customize the metrics logged by using your own surrogate models or critic metrics.\n", + "* Check out the [examples](https://github.com/whylabs/langkit/tree/main/langkit/examples) folder for scenarios from [\"Hello World!\"](https://github.com/whylabs/langkit/blob/main/langkit/examples/Logging_Text.ipynb) to [monitoring an LLM](https://github.com/whylabs/langkit/blob/main/langkit/examples/LLM_to_WhyLabs.ipynb) in production!\n", + "* Learn more about the [features](https://github.com/whylabs/langkit#features-%EF%B8%8F) LangKit extracts out of the box.\n", + "* Learn more about LangKit's [modules documentation](https://github.com/whylabs/langkit/blob/main/langkit/docs/modules.md).\n", + "* Explore more on [WhyLabs](https://whylabs.ai/safeguard-large-language-models?utm_source=github&utm_medium=referral&utm_campaign=langkit) and monitor your LLM application over time!\n", + "\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.11.2 64-bit", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.10" + }, + "orig_nbformat": 4, + "vscode": { + "interpreter": { + "hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e" + } + }, + "colab": { + "provenance": [] } - ], - "source": [ - "pos_chats = load_chats(\"pos\")\n", - "show_first_chat(pos_chats)\n", - "\n", - "neg_chats = load_chats(\"neg\")\n", - "show_first_chat(neg_chats)\n", - "\n", - "results_comparison = why.log(multiple={\"positive_chats\": pos_chats,\n", - " \"negative_chats\": neg_chats},\n", - " schema=schema)" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In this example we are using anonymous guest sessions to view the collected metrics in WhyLabs, You can explore and compare specific metrics, in this example we expect a large and obvious distribution drift in the sentiment scores on the response which you can see [here](https://hub.whylabsapp.com/resources/model-1/profiles?feature-highlight=response.sentiment_nltk&includeType=discrete&includeType=non-discrete&limit=30&offset=0&sessionToken=session-8gcsnbVy&sortModelBy=LatestAlert&sortModelDirection=DESC&profile=ref-ONdpltPp9jno6qMM&profile=ref-HoQEiEKtaaAvGG7S).\n" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "### ๐Ÿ“ˆ Monitor Metrics over time\n", - "\n", - "\n", - "WhyLabs hosts a public demo org where we show a test LLM application being monitored over time. With LangKit telemetry collected over time, you can detect changes systematically and be alerted to potential problems in a deployed LLM application. We think a variety of different metrics can be helpful in detecting anamolies. In this example, something has triggered a series of alerts. Can you guess what went wrong? \n", - "\n", - "##### ๐ŸŽฏ **Explore a demo environment** [here](https://hub.whylabsapp.com/resources/demo-llm-chatbot/columns/prompt.sentiment_nltk?dateRange=2023-06-08-to-2023-06-09&sortModelBy=LatestAlert&sortModelDirection=DESC&targetOrgId=demo&sessionToken=session-8gcsnbVy)!\n", - "\n", - "\n", - "### โœ… Next Steps\n", - "If you see value in detecting changes in how your LLM application is behaving, you might take a look at some of our other examples showing how to monitor these metrics as a timeseries for an LLM application in production, or how to customize the metrics logged by using your own surrogate models or critic metrics.\n", - "* Check out the [examples](https://github.com/whylabs/langkit/tree/main/langkit/examples) folder for scenarios from [\"Hello World!\"](https://github.com/whylabs/langkit/blob/main/langkit/examples/Logging_Text.ipynb) to [monitoring an LLM](https://github.com/whylabs/langkit/blob/main/langkit/examples/LLM_to_WhyLabs.ipynb) in production!\n", - "* Learn more about the [features](https://github.com/whylabs/langkit#features-%EF%B8%8F) LangKit extracts out of the box.\n", - "* Learn more about LangKit's [modules documentation](https://github.com/whylabs/langkit/blob/main/langkit/docs/modules.md).\n", - "* Explore more on [WhyLabs](https://whylabs.ai/safeguard-large-language-models?utm_source=github&utm_medium=referral&utm_campaign=langkit) and monitor your LLM application over time!\n", - "\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.11.2 64-bit", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.8.10" }, - "orig_nbformat": 4, - "vscode": { - "interpreter": { - "hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e" - } - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} + "nbformat": 4, + "nbformat_minor": 0 +} \ No newline at end of file From fe1fd2e36ea1aca30f8d9558d86ddf7542d407ad Mon Sep 17 00:00:00 2001 From: richard-rogers <93153899+richard-rogers@users.noreply.github.com> Date: Mon, 30 Sep 2024 15:10:39 +0100 Subject: [PATCH 2/7] Update huggingface_langkit_whylabs.ipynb --- langkit/examples/huggingface_langkit_whylabs.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/langkit/examples/huggingface_langkit_whylabs.ipynb b/langkit/examples/huggingface_langkit_whylabs.ipynb index 7637349c..e14bce03 100644 --- a/langkit/examples/huggingface_langkit_whylabs.ipynb +++ b/langkit/examples/huggingface_langkit_whylabs.ipynb @@ -144,7 +144,7 @@ "from langkit import llm_metrics # alternatively use 'light_metrics'\n", "import whylogs as why\n", "\n", - "why.init(session_type='whylabs_anonymous')\n", + "why.init(upload_on_log=True)\n", "# Note: llm_metrics.init() downloads models so this is slow first time.\n", "schema = llm_metrics.init()" ] From 4a1b0c50ca137c38f949d4773d672c9c78ab157b Mon Sep 17 00:00:00 2001 From: Richard Rogers Date: Mon, 30 Sep 2024 15:30:35 +0100 Subject: [PATCH 3/7] pre-commit --- langkit/examples/Intro_to_Langkit.ipynb | 466 ++++++++++++------------ 1 file changed, 226 insertions(+), 240 deletions(-) diff --git a/langkit/examples/Intro_to_Langkit.ipynb b/langkit/examples/Intro_to_Langkit.ipynb index 2e04e932..b323e2bd 100644 --- a/langkit/examples/Intro_to_Langkit.ipynb +++ b/langkit/examples/Intro_to_Langkit.ipynb @@ -1,253 +1,239 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "id": "8XJSw3fgclIZ" - }, - "source": [ - "\n", - "## ๐Ÿ“– Intro to LangKit\n", - "\n", - "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/whylabs/langkit/blob/main/langkit/examples/Intro_to_Langkit.ipynb)\n", - "\n", - "\n", - "\n", - "Table of Contents\n", - "- [Intro to LangKit](#intro-to-langkit)\n", - "- [What is LangKit](#what-is-langkit)\n", - "- [Initialize LLM metrics](#initialize-llm-metrics)\n", - "- [Hello, World!](#hello-world)\n", - "- [Comparing Data](#comparing-data)\n", - "- [Monitor Metrics over time](#monitor-metrics-over-time)\n", - "- [Next Steps](#next-steps)\n", - "\n", - "\n", - "### ๐Ÿ“š What is LangKit?\n", - ">LangKit is an open-source text metrics toolkit for monitoring language models. It offers an array of methods for extracting relevant signals from the input and/or output text, which are compatible with the open-source data logging library [whylogs](https://whylogs.readthedocs.io/en/latest).\n", - ">In this example, we'll look for distribution drift in the sentiment scores on the model response.๐Ÿ’ก\n", - "\n", - "Ok! let's install __langkit__." - ] - }, - { - "cell_type": "code", - "source": [ - "%pip install ./whylogs-1.4.8-py3-none-any.whl" - ], - "metadata": { - "collapsed": true, - "id": "2-QsnYlXc85k" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": true, - "id": "fiNCAytMclIe" - }, - "outputs": [], - "source": [ - "# Note: you may need to restart the kernel to use updated packages.\n", - "%pip install -U langkit[all]" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "3KZhfMlaclIg" - }, - "source": [ - "\n", - "### ๐Ÿš€ Initialize LLM metrics\n", - "LangKit provides a toolkit of metrics for LLM applications, lets initialize them!" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "ub53VXDAclIg" - }, - "outputs": [], - "source": [ - "from langkit import llm_metrics # alternatively use 'light_metrics'\n", - "import whylogs as why\n", - "\n", - "why.init(upload_on_log=True)\n", - "# Note: llm_metrics.init() downloads models so this is slow first time.\n", - "schema = llm_metrics.init()" - ] - }, + "cells": [ + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "## ๐Ÿ“– Intro to LangKit\n", + "\n", + "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/whylabs/langkit/blob/main/langkit/examples/Intro_to_Langkit.ipynb)\n", + "\n", + "\n", + "\n", + "Table of Contents\n", + "- [Intro to LangKit](#intro-to-langkit)\n", + "- [What is LangKit](#what-is-langkit)\n", + "- [Initialize LLM metrics](#initialize-llm-metrics)\n", + "- [Hello, World!](#hello-world)\n", + "- [Comparing Data](#comparing-data)\n", + "- [Monitor Metrics over time](#monitor-metrics-over-time)\n", + "- [Next Steps](#next-steps)\n", + "\n", + "\n", + "### ๐Ÿ“š What is LangKit?\n", + ">LangKit is an open-source text metrics toolkit for monitoring language models. It offers an array of methods for extracting relevant signals from the input and/or output text, which are compatible with the open-source data logging library [whylogs](https://whylogs.readthedocs.io/en/latest).\n", + ">In this example, we'll look for distribution drift in the sentiment scores on the model response.๐Ÿ’ก\n", + "\n", + "Ok! let's install __langkit__." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Note: you may need to restart the kernel to use updated packages.\n", + "%pip install -U langkit[all]" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### ๐Ÿš€ Initialize LLM metrics\n", + "LangKit provides a toolkit of metrics for LLM applications, lets initialize them!" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from langkit import llm_metrics # alternatively use 'light_metrics'\n", + "import whylogs as why\n", + "\n", + "why.init(upload_on_log=True)\n", + "# Note: llm_metrics.init() downloads models so this is slow first time.\n", + "schema = llm_metrics.init()" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### ๐Ÿ‘‹ Hello, World!\n", + "In the below code we log a few example prompt/response pairs and send metrics to WhyLabs." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ { - "cell_type": "markdown", - "metadata": { - "id": "mhFiRDVPclIh" - }, - "source": [ - "\n", - "### ๐Ÿ‘‹ Hello, World!\n", - "In the below code we log a few example prompt/response pairs and send metrics to WhyLabs." - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "There are 50 records in this toy example data, here's the first one:\n", + "prompt: Hello, response: World!\n", + "\n", + "โœ… Aggregated 50 rows into profile 'langkit-sample-chats-all'\n", + "\n", + "Visualize and explore this profile with one-click\n" + ] }, { - "cell_type": "code", - "execution_count": 5, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "aGRTORs3clIh", - "outputId": "eed1654a-8198-495b-9a06-2a8b81873fd8" - }, - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "There are 50 records in this toy example data, here's the first one:\n", - "prompt: Hello, response: World!\n", - "\n", - "\n", - "โœ… Aggregated 50 rows into profile langkit-sample-chats-all\n", - "\n", - "Visualize and explore this profile with one-click\n", - "๐Ÿ” https://hub.whylabsapp.com/resources/model-1/profiles?profile=ref-7ekFkHbGm0KS9pGR&sessionToken=session-sfnxLEmy\n" - ] - } - ], - "source": [ - "from langkit.whylogs.samples import load_chats, show_first_chat\n", - "\n", - "# Let's look at what's in this toy example:\n", - "chats = load_chats()\n", - "print(f\"There are {len(chats)} records in this toy example data, here's the first one:\")\n", - "show_first_chat(chats)\n", - "\n", - "results = why.log(chats, name=\"langkit-sample-chats-all\", schema=schema)" - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ” https://hub.whylabsapp.com/resources/model-1/profiles?sessionToken=session-8gcsnbVy&profile=ref-AVbB1yOaSblsa89U\n" + ] }, { - "cell_type": "markdown", - "metadata": { - "id": "Nya78_R5clIj" - }, - "source": [ - "\n", - "### ๐Ÿ” Comparing Data\n", - "Things get more interesting when you can compare two sets of metrics from an LLM application. The power of gathering systematic telemetry over time comes from being able to see how these metrics change, or how two sets of profiles compare. Below we asked GPT for some positive words and then asked for negative words as part of these two toy examples.\n", - "\n", - "> ๐Ÿ’ก Take a look at the difference in the `response.sentiment_nltk` distributions between the two profiles. The first example is much more positive than the second example." - ] - }, + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "source": [ + "from langkit.whylogs.samples import load_chats, show_first_chat\n", + "\n", + "# Let's look at what's in this toy example:\n", + "chats = load_chats()\n", + "print(f\"There are {len(chats)} records in this toy example data, here's the first one:\")\n", + "show_first_chat(chats)\n", + "\n", + "results = why.log(chats, name=\"langkit-sample-chats-all\", schema=schema)" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### ๐Ÿ” Comparing Data\n", + "Things get more interesting when you can compare two sets of metrics from an LLM application. The power of gathering systematic telemetry over time comes from being able to see how these metrics change, or how two sets of profiles compare. Below we asked GPT for some positive words and then asked for negative words as part of these two toy examples.\n", + "\n", + "> ๐Ÿ’ก Take a look at the difference in the `response.sentiment_nltk` distributions between the two profiles. The first example is much more positive than the second example." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ { - "cell_type": "code", - "execution_count": 6, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "SCjn-xRYclIj", - "outputId": "3f6e4042-1736-4870-9cf9-013f8c5ef270" - }, - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "prompt: What do you think about puppies? response: Puppies are absolutely adorable. Their playful nature and boundless energy can bring a lot of joy and happiness.\n", - "\n", - "prompt: Can you describe a difficult day? response: A difficult day might be filled with challenging tasks, stressful situations, and unexpected obstacles. :-( These moments can feel overwhelming and can lead to feelings of frustration!\n", - "\n", - "\n", - "โœ… Aggregated 14 lines into profile 'positive_chats', 20 lines into profile 'negative_chats'\n", - "\n", - "Visualize and explore the profiles with one-click\n", - "๐Ÿ” https://hub.whylabsapp.com/resources/model-1/profiles?profile=ref-fRlQkKmljN1WVbeA&profile=ref-eBJDmruQVMALJZ80&sessionToken=session-sfnxLEmy\n", - "\n", - "Or view each profile individually\n", - " โคท https://hub.whylabsapp.com/resources/model-1/profiles?profile=ref-fRlQkKmljN1WVbeA&sessionToken=session-sfnxLEmy\n", - " โคท https://hub.whylabsapp.com/resources/model-1/profiles?profile=ref-eBJDmruQVMALJZ80&sessionToken=session-sfnxLEmy\n" - ] - } - ], - "source": [ - "pos_chats = load_chats(\"pos\")\n", - "show_first_chat(pos_chats)\n", - "\n", - "neg_chats = load_chats(\"neg\")\n", - "show_first_chat(neg_chats)\n", - "\n", - "results_comparison = why.log(multiple={\"positive_chats\": pos_chats,\n", - " \"negative_chats\": neg_chats},\n", - " schema=schema)" - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "prompt: What do you think about puppies? response: Puppies are absolutely adorable. Their playful nature and boundless energy can bring a lot of joy and happiness.\n", + "\n", + "prompt: Can you describe a difficult day? response: A difficult day might be filled with challenging tasks, stressful situations, and unexpected obstacles. :-( These moments can feel overwhelming and can lead to feelings of frustration!\n", + "\n", + "โœ… Aggregated 14 lines into profile 'positive_chats', 20 lines into profile 'negative_chats'\n", + "\n", + "Visualize and explore the profiles with one-click\n" + ] }, { - "cell_type": "markdown", - "metadata": { - "id": "FkQOaqEFclIk" - }, - "source": [ - "In this example we are using anonymous guest sessions to view the collected metrics in WhyLabs, You can explore and compare specific metrics, in this example we expect a large and obvious distribution drift in the sentiment scores on the response which you can see [here](https://hub.whylabsapp.com/resources/model-1/profiles?feature-highlight=response.sentiment_nltk&includeType=discrete&includeType=non-discrete&limit=30&offset=0&sessionToken=session-8gcsnbVy&sortModelBy=LatestAlert&sortModelDirection=DESC&profile=ref-ONdpltPp9jno6qMM&profile=ref-HoQEiEKtaaAvGG7S).\n" - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "๐Ÿ” https://hub.whylabsapp.com/resources/model-1/profiles?sessionToken=session-8gcsnbVy&profile=ref-ONdpltPp9jno6qMM&profile=ref-HoQEiEKtaaAvGG7S\n", + "\n", + "Or view each profile individually\n", + " โคท https://hub.whylabsapp.com/resources/model-1/profiles?session-8gcsnbVy&profile=ref-ONdpltPp9jno6qMM\n", + " โคท https://hub.whylabsapp.com/resources/model-1/profiles?session-8gcsnbVy&profile=ref-HoQEiEKtaaAvGG7S\n" + ] }, { - "cell_type": "markdown", - "metadata": { - "id": "9qotNVoOclIk" - }, - "source": [ - "\n", - "### ๐Ÿ“ˆ Monitor Metrics over time\n", - "\n", - "\n", - "WhyLabs hosts a public demo org where we show a test LLM application being monitored over time. With LangKit telemetry collected over time, you can detect changes systematically and be alerted to potential problems in a deployed LLM application. We think a variety of different metrics can be helpful in detecting anamolies. In this example, something has triggered a series of alerts. Can you guess what went wrong?\n", - "\n", - "##### ๐ŸŽฏ **Explore a demo environment** [here](https://hub.whylabsapp.com/resources/demo-llm-chatbot/columns/prompt.sentiment_nltk?dateRange=2023-06-08-to-2023-06-09&sortModelBy=LatestAlert&sortModelDirection=DESC&targetOrgId=demo&sessionToken=session-8gcsnbVy)!\n", - "\n", - "\n", - "### โœ… Next Steps\n", - "If you see value in detecting changes in how your LLM application is behaving, you might take a look at some of our other examples showing how to monitor these metrics as a timeseries for an LLM application in production, or how to customize the metrics logged by using your own surrogate models or critic metrics.\n", - "* Check out the [examples](https://github.com/whylabs/langkit/tree/main/langkit/examples) folder for scenarios from [\"Hello World!\"](https://github.com/whylabs/langkit/blob/main/langkit/examples/Logging_Text.ipynb) to [monitoring an LLM](https://github.com/whylabs/langkit/blob/main/langkit/examples/LLM_to_WhyLabs.ipynb) in production!\n", - "* Learn more about the [features](https://github.com/whylabs/langkit#features-%EF%B8%8F) LangKit extracts out of the box.\n", - "* Learn more about LangKit's [modules documentation](https://github.com/whylabs/langkit/blob/main/langkit/docs/modules.md).\n", - "* Explore more on [WhyLabs](https://whylabs.ai/safeguard-large-language-models?utm_source=github&utm_medium=referral&utm_campaign=langkit) and monitor your LLM application over time!\n", - "\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.11.2 64-bit", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.8.10" - }, - "orig_nbformat": 4, - "vscode": { - "interpreter": { - "hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e" - } - }, - "colab": { - "provenance": [] + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] } + ], + "source": [ + "pos_chats = load_chats(\"pos\")\n", + "show_first_chat(pos_chats)\n", + "\n", + "neg_chats = load_chats(\"neg\")\n", + "show_first_chat(neg_chats)\n", + "\n", + "results_comparison = why.log(multiple={\"positive_chats\": pos_chats,\n", + " \"negative_chats\": neg_chats},\n", + " schema=schema)" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In this example we are using anonymous guest sessions to view the collected metrics in WhyLabs, You can explore and compare specific metrics, in this example we expect a large and obvious distribution drift in the sentiment scores on the response which you can see [here](https://hub.whylabsapp.com/resources/model-1/profiles?feature-highlight=response.sentiment_nltk&includeType=discrete&includeType=non-discrete&limit=30&offset=0&sessionToken=session-8gcsnbVy&sortModelBy=LatestAlert&sortModelDirection=DESC&profile=ref-ONdpltPp9jno6qMM&profile=ref-HoQEiEKtaaAvGG7S).\n" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### ๐Ÿ“ˆ Monitor Metrics over time\n", + "\n", + "\n", + "WhyLabs hosts a public demo org where we show a test LLM application being monitored over time. With LangKit telemetry collected over time, you can detect changes systematically and be alerted to potential problems in a deployed LLM application. We think a variety of different metrics can be helpful in detecting anamolies. In this example, something has triggered a series of alerts. Can you guess what went wrong? \n", + "\n", + "##### ๐ŸŽฏ **Explore a demo environment** [here](https://hub.whylabsapp.com/resources/demo-llm-chatbot/columns/prompt.sentiment_nltk?dateRange=2023-06-08-to-2023-06-09&sortModelBy=LatestAlert&sortModelDirection=DESC&targetOrgId=demo&sessionToken=session-8gcsnbVy)!\n", + "\n", + "\n", + "### โœ… Next Steps\n", + "If you see value in detecting changes in how your LLM application is behaving, you might take a look at some of our other examples showing how to monitor these metrics as a timeseries for an LLM application in production, or how to customize the metrics logged by using your own surrogate models or critic metrics.\n", + "* Check out the [examples](https://github.com/whylabs/langkit/tree/main/langkit/examples) folder for scenarios from [\"Hello World!\"](https://github.com/whylabs/langkit/blob/main/langkit/examples/Logging_Text.ipynb) to [monitoring an LLM](https://github.com/whylabs/langkit/blob/main/langkit/examples/LLM_to_WhyLabs.ipynb) in production!\n", + "* Learn more about the [features](https://github.com/whylabs/langkit#features-%EF%B8%8F) LangKit extracts out of the box.\n", + "* Learn more about LangKit's [modules documentation](https://github.com/whylabs/langkit/blob/main/langkit/docs/modules.md).\n", + "* Explore more on [WhyLabs](https://whylabs.ai/safeguard-large-language-models?utm_source=github&utm_medium=referral&utm_campaign=langkit) and monitor your LLM application over time!\n", + "\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.11.2 64-bit", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.10" }, - "nbformat": 4, - "nbformat_minor": 0 -} \ No newline at end of file + "orig_nbformat": 4, + "vscode": { + "interpreter": { + "hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e" + } + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} From 1ceae500e82821d9a90092d4e065415118a432bb Mon Sep 17 00:00:00 2001 From: Richard Rogers Date: Tue, 1 Oct 2024 18:07:20 +0100 Subject: [PATCH 4/7] dependency version bump --- pyproject.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pyproject.toml b/pyproject.toml index a2b4a638..7c07e1fc 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -11,7 +11,7 @@ readme = "DESCRIPTION.md" [tool.poetry.dependencies] python = ">=3.8,<4" pandas = "*" -whylogs = "^1.3.19" +whylogs = "^1.5.0" # optional dependencies From 529cce5fcf89ff291bc2abf97cd82a6d3809cfac Mon Sep 17 00:00:00 2001 From: Richard Rogers Date: Tue, 8 Oct 2024 21:49:11 +0000 Subject: [PATCH 5/7] poetry.lock update --- poetry.lock | 53 +++++++++++++++++++++++++---------------------------- 1 file changed, 25 insertions(+), 28 deletions(-) diff --git a/poetry.lock b/poetry.lock index 862b3ed1..58c9c357 100644 --- a/poetry.lock +++ b/poetry.lock @@ -216,6 +216,17 @@ files = [ {file = "backcall-0.2.0.tar.gz", hash = "sha256:5cbdbf27be5e7cfadb448baf0aa95508f91f2bbc6c6437cd9cd06e2a4c215e1e"}, ] +[[package]] +name = "backoff" +version = "2.2.1" +description = "Function decoration for backoff and retry" +optional = false +python-versions = ">=3.7,<4.0" +files = [ + {file = "backoff-2.2.1-py3-none-any.whl", hash = "sha256:63579f9a0628e06278f7e47b7d7d5b6ce20dc65c5e96a6f3ca99a6adca0396e8"}, + {file = "backoff-2.2.1.tar.gz", hash = "sha256:03f829f5bb1923180821643f8753b0502c3b682293992485b0eef2807afa5cba"}, +] + [[package]] name = "black" version = "23.12.1" @@ -4080,20 +4091,6 @@ dev = ["autoflake (>=1.3.1,<2.0.0)", "flake8 (>=3.8.3,<4.0.0)", "pre-commit (>=2 doc = ["cairosvg (>=2.5.2,<3.0.0)", "mdx-include (>=1.4.1,<2.0.0)", "mkdocs (>=1.1.2,<2.0.0)", "mkdocs-material (>=8.1.4,<9.0.0)", "pillow (>=9.3.0,<10.0.0)"] test = ["black (>=22.3.0,<23.0.0)", "coverage (>=6.2,<7.0)", "isort (>=5.0.6,<6.0.0)", "mypy (==0.910)", "pytest (>=4.4.0,<8.0.0)", "pytest-cov (>=2.10.0,<5.0.0)", "pytest-sugar (>=0.9.4,<0.10.0)", "pytest-xdist (>=1.32.0,<4.0.0)", "rich (>=10.11.0,<14.0.0)", "shellingham (>=1.3.0,<2.0.0)"] -[[package]] -name = "types-requests" -version = "2.31.0.20240218" -description = "Typing stubs for requests" -optional = false -python-versions = ">=3.8" -files = [ - {file = "types-requests-2.31.0.20240218.tar.gz", hash = "sha256:f1721dba8385958f504a5386240b92de4734e047a08a40751c1654d1ac3349c5"}, - {file = "types_requests-2.31.0.20240218-py3-none-any.whl", hash = "sha256:a82807ec6ddce8f00fe0e949da6d6bc1fbf1715420218a9640d695f70a9e5a9b"}, -] - -[package.dependencies] -urllib3 = ">=2" - [[package]] name = "types-setuptools" version = "69.1.0.20240302" @@ -4242,13 +4239,13 @@ wasabi = ">=0.9.1,<1.2.0" [[package]] name = "whylabs-client" -version = "0.5.10" +version = "0.6.10" description = "WhyLabs API client" optional = false python-versions = ">=3.6" files = [ - {file = "whylabs-client-0.5.10.tar.gz", hash = "sha256:267e5ef9e5811953e83ccd5dbe4ece21b4da2aa26adcc851b95da104c1f4cf7c"}, - {file = "whylabs_client-0.5.10-py3-none-any.whl", hash = "sha256:41a87e973e6e4410bc33e7f13d2cdd40ac4787f060cbd973ab16828cb144fd35"}, + {file = "whylabs-client-0.6.10.tar.gz", hash = "sha256:55328d8606ae2e5466fa1be7db3333f8973ac8002edcfd75b14b4acce0110dfb"}, + {file = "whylabs_client-0.6.10-py3-none-any.whl", hash = "sha256:d1aa38f57493c11d503210674790634194f6126a915cb7b18576542fa9a55198"}, ] [package.dependencies] @@ -4272,38 +4269,38 @@ syllapy = "*" [[package]] name = "whylogs" -version = "1.3.24" +version = "1.5.0" description = "Profile and monitor your ML data pipeline end-to-end" optional = false -python-versions = ">=3.7.1,<4" +python-versions = "<4,>=3.7.1" files = [ - {file = "whylogs-1.3.24-py3-none-any.whl", hash = "sha256:27c434bb55e62c4cf0432f55b4b8202cbc6e847d350a86659201a0b574692a23"}, - {file = "whylogs-1.3.24.tar.gz", hash = "sha256:1e0459e5a0e7adc5eba9407192a7adf8de3bad91646afe638b48726e94437a28"}, + {file = "whylogs-1.5.0-py3-none-any.whl", hash = "sha256:33afb99c8abdf8758cf97dfe2bc2c9029d286bbbb29ddcd1ebfbdd18efe4959d"}, + {file = "whylogs-1.5.0.tar.gz", hash = "sha256:4c48846a0333b97813cf81f676396bd28c33954a86ef5f4e42e12585f6e4393d"}, ] [package.dependencies] +backoff = ">=2.2.1,<3.0.0" platformdirs = ">=3.5.0,<4.0.0" protobuf = ">=3.19.4" requests = ">=2.27,<3.0" -types-requests = ">=2.30.0.0,<3.0.0.0" typing-extensions = {version = ">=3.10", markers = "python_version < \"4\""} -whylabs-client = ">=0.5.10,<0.6.0" +whylabs-client = ">=0.6.5,<0.7.0" whylogs-sketching = ">=3.4.1.dev3" [package.extras] -all = ["Pillow (>=9.2.0,<10.0.0)", "boto3 (>=1.22.13,<2.0.0)", "faster-fifo (>=1.4.5,<2.0.0)", "fugue (>=0.8.1,<0.9.0)", "google-cloud-storage (>=2.5.0,<3.0.0)", "ipython", "mlflow-skinny (<2.0.1)", "mlflow-skinny (>=2.5.0,<3.0.0)", "numpy", "numpy (>=1.23.2)", "orjson (>=3.8.10,<4.0.0)", "pandas", "pyarrow (>=8.0.0,<13)", "pybars3 (>=0.9,<0.10)", "pyspark (>=3.0.0,<4.0.0)", "scikit-learn (>=1.0.2,<2.0.0)", "scikit-learn (>=1.1.2,<2)", "scipy (>=1.5)", "scipy (>=1.9.2)"] +all = ["Pillow (>=10.1.0,<11)", "Pillow (>=9.2.0)", "boto3 (>=1.22.13,<2.0.0)", "faster-fifo (>=1.4.5,<2.0.0)", "fugue (>=0.8.1,<0.9.0)", "google-cloud-storage (>=2.5.0,<3.0.0)", "ipython", "mlflow-skinny (<2.0.1)", "mlflow-skinny (>=2.5.0,<3.0.0)", "numpy", "numpy (>=1.23.2)", "orjson (>=3.8.10,<4.0.0)", "pandas", "pyarrow (>=8.0.0,<13)", "pybars3 (>=0.9,<0.10)", "pyspark (>=3.0.0,<4.0.0)", "scikit-learn (>=1.0.2,<2.0.0)", "scikit-learn (>=1.1.2,<2)", "scipy (>=1.5)", "scipy (>=1.9.2)"] datasets = ["pandas"] docs = ["furo (>=2022.3.4,<2023.0.0)", "ipython_genutils (>=0.2.0,<0.3.0)", "myst-parser[sphinx] (>=0.17.2,<0.18.0)", "nbconvert (>=7.0.0,<8.0.0)", "nbsphinx (>=0.8.9,<0.9.0)", "sphinx", "sphinx-autoapi", "sphinx-autobuild (>=2021.3.14,<2022.0.0)", "sphinx-copybutton (>=0.5.0,<0.6.0)", "sphinx-inline-tabs", "sphinxext-opengraph (>=0.6.3,<0.7.0)"] embeddings = ["numpy", "numpy (>=1.23.2)", "scikit-learn (>=1.0.2,<2.0.0)", "scikit-learn (>=1.1.2,<2)"] fugue = ["fugue (>=0.8.1,<0.9.0)"] gcs = ["google-cloud-storage (>=2.5.0,<3.0.0)"] -image = ["Pillow (>=9.2.0,<10.0.0)", "numpy", "numpy (>=1.23.2)"] -mlflow = ["mlflow-skinny (<2.0.1)", "mlflow-skinny (>=2.5.0,<3.0.0)"] +image = ["Pillow (>=10.1.0,<11)", "Pillow (>=9.2.0)", "numpy", "numpy (>=1.23.2)"] +mlflow = ["databricks-cli (>=0.8.0,<0.9.0)", "mlflow-skinny (<2.0.1)", "mlflow-skinny (>=2.5.0,<3.0.0)"] proc = ["faster-fifo (>=1.4.5,<2.0.0)", "orjson (>=3.8.10,<4.0.0)", "pandas"] proc-mp = ["orjson (>=3.8.10,<4.0.0)", "pandas"] s3 = ["boto3 (>=1.22.13,<2.0.0)"] spark = ["pyarrow (>=8.0.0,<13)", "pyspark (>=3.0.0,<4.0.0)"] -viz = ["Pillow (>=9.2.0,<10.0.0)", "ipython", "numpy", "numpy (>=1.23.2)", "pybars3 (>=0.9,<0.10)", "scipy (>=1.5)", "scipy (>=1.9.2)"] +viz = ["Pillow (>=10.1.0,<11)", "Pillow (>=9.2.0)", "ipython", "numpy", "numpy (>=1.23.2)", "pybars3 (>=0.9,<0.10)", "scipy (>=1.5)", "scipy (>=1.9.2)"] [[package]] name = "whylogs-sketching" @@ -4613,4 +4610,4 @@ all = ["datasets", "detoxify", "evaluate", "h5py", "ipywidgets", "nltk", "numpy" [metadata] lock-version = "2.0" python-versions = ">=3.8,<4" -content-hash = "493319b926308e139050ac60eb0692bef12ee43e2e2a4549a34de13e10639a58" +content-hash = "fc3375b5f7756337fb04d6f69cc3a70645991ee369db925f0b0968ed6d222cde" From 7f7a5a3b84406adc1288a658705685c6ca783a03 Mon Sep 17 00:00:00 2001 From: richard-rogers <93153899+richard-rogers@users.noreply.github.com> Date: Tue, 8 Oct 2024 23:13:27 +0100 Subject: [PATCH 6/7] Update Intro_to_Langkit.ipynb --- langkit/examples/Intro_to_Langkit.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/langkit/examples/Intro_to_Langkit.ipynb b/langkit/examples/Intro_to_Langkit.ipynb index b323e2bd..2dbc2c30 100644 --- a/langkit/examples/Intro_to_Langkit.ipynb +++ b/langkit/examples/Intro_to_Langkit.ipynb @@ -36,7 +36,7 @@ "outputs": [], "source": [ "# Note: you may need to restart the kernel to use updated packages.\n", - "%pip install -U langkit[all]" + "%pip install -U 'langkit[all]>=0.0.34'" ] }, { From d14734503d1365f104ec253e6088c4403b83b3d4 Mon Sep 17 00:00:00 2001 From: richard-rogers <93153899+richard-rogers@users.noreply.github.com> Date: Tue, 8 Oct 2024 23:18:42 +0100 Subject: [PATCH 7/7] Update huggingface_langkit_whylabs.ipynb --- langkit/examples/huggingface_langkit_whylabs.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/langkit/examples/huggingface_langkit_whylabs.ipynb b/langkit/examples/huggingface_langkit_whylabs.ipynb index e14bce03..fd1ae0d3 100644 --- a/langkit/examples/huggingface_langkit_whylabs.ipynb +++ b/langkit/examples/huggingface_langkit_whylabs.ipynb @@ -42,7 +42,7 @@ "outputs": [], "source": [ "%pip install transformers\n", - "%pip install 'langkit[all]'" + "%pip install 'langkit[all]>=0.0.34'" ] }, {