D1_07_PEFT.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2c272574-6a76-40c2-b7fd-9cadce7ef1c5",
   "metadata": {},
   "source": [
    "## Parameter-Efficient Fine-Tuning (PEFT)\n",
    "\n",
    "Fine-tuning large pre-trained language models can be computationally expensive and require significant memory, especially when all model parameters are updated during training. **Parameter-Efficient Fine-Tuning (PEFT)** techniques aim to reduce the number of trainable parameters, allowing for efficient fine-tuning with less memory and computational cost.\n",
    "\n",
    "PEFT is especially useful if the pre-trained model is loaded as a quantized model to save GPU memory and thus cannot be trained.\n",
    "\n",
    "In this example, we will fine-tune a pre-trained model using **Low-Rank Adapters (LoRA)** to demonstrate parameter-efficient fine-tuning.\n",
    "\n",
    "<!-- https://www.urlencoder.org -->\n",
    "<!-- https://base64.guru/converter/encode/image -->\n",
    "![alt text](data:image/svg+xml,%3Csvg%20width%3D%22762%22%20height%3D%22375%22%20viewBox%3D%220%200%201089%20535%22%20fill%3D%22none%22%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%3E%3Cg%20clip-path%3D%22url%28%23clip0_1_194%29%22%3E%3Cpath%20d%3D%22M35.4933%20221.568H74.5067V224.639H35.4933V221.568ZM35.4933%20231.964H74.5067V235.036H35.4933V231.964ZM35.4933%20242.358H62.7778V245.429H35.4933V242.358ZM32.0276%20212C30.5964%20212%2029.4007%20212.473%2028.4404%20213.419C27.4801%20214.365%2027%20215.546%2027%20216.963V250.037C27%20251.451%2027.4801%20252.632%2028.4404%20253.578C29.4007%20254.524%2030.5954%20254.998%2032.0244%20255H77.9756C79.4067%20255%2080.6013%20254.526%2081.5596%20253.578C82.5178%20252.63%2082.9979%20251.449%2083%20250.037V216.96C83%20215.548%2082.5199%20214.367%2081.5596%20213.419C80.5993%20212.471%2079.4046%20211.998%2077.9756%20212H32.0276ZM32.0276%20215.071H77.9787C78.4557%20215.071%2078.8944%20215.268%2079.2947%20215.661C79.695%20216.054%2079.8941%20216.488%2079.892%20216.963V250.037C79.892%20250.51%2079.6929%20250.944%2079.2947%20251.339C78.8964%20251.734%2078.4567%20251.931%2077.9756%20251.929H32.0244C31.5474%20251.929%2031.1087%20251.732%2030.7084%20251.339C30.3081%20250.946%2030.109%20250.512%2030.1111%20250.037V216.96C30.1111%20216.489%2030.3102%20216.056%2030.7084%20215.661C31.1067%20215.266%2031.5453%20215.069%2032.0244%20215.071%22%20fill%3D%22%238D8D8D%22%2F%3E%3Cpath%20d%3D%22M32.0276%20215.071H77.9787C78.4557%20215.071%2078.8944%20215.268%2079.2947%20215.661C79.695%20216.054%2079.8941%20216.488%2079.892%20216.963V250.037C79.892%20250.51%2079.6929%20250.944%2079.2947%20251.339C78.8964%20251.734%2078.4567%20251.931%2077.9756%20251.929H32.0244C31.5474%20251.929%2031.1087%20251.732%2030.7084%20251.339C30.3081%20250.946%2030.109%20250.512%2030.1111%20250.037V216.96C30.1111%20216.489%2030.3102%20216.056%2030.7084%20215.661C31.1067%20215.266%2031.5453%20215.069%2032.0244%20215.071M30.1111%20215.071V251.929M35.4933%20221.568H74.5067V224.639H35.4933V221.568ZM35.4933%20231.964H74.5067V235.036H35.4933V231.964ZM35.4933%20242.358H62.7778V245.429H35.4933V242.358ZM32.0276%20212C30.5964%20212%2029.4007%20212.473%2028.4404%20213.419C27.4801%20214.365%2027%20215.546%2027%20216.963V250.037C27%20251.451%2027.4801%20252.632%2028.4404%20253.578C29.4007%20254.524%2030.5954%20254.998%2032.0244%20255H77.9756C79.4067%20255%2080.6013%20254.526%2081.5596%20253.578C82.5178%20252.63%2082.9979%20251.449%2083%20250.037V216.96C83%20215.548%2082.5199%20214.367%2081.5596%20213.419C80.5993%20212.471%2079.4046%20211.998%2077.9756%20212H32.0276Z%22%20stroke%3D%22%23F4F4F7%22%2F%3E%3Crect%20x%3D%22329.5%22%20y%3D%2227.5%22%20width%3D%22419%22%20height%3D%22204%22%20rx%3D%2219.5%22%20fill%3D%22%23FCFCFC%22%20stroke%3D%22%23D9D9D9%22%2F%3E%3Crect%20x%3D%22330.5%22%20y%3D%22280.5%22%20width%3D%22417%22%20height%3D%22232%22%20rx%3D%2218.5%22%20stroke%3D%22%23D9D9D9%22%20stroke-width%3D%223%22%20stroke-dasharray%3D%2210%2010%22%2F%3E%3Cmask%20id%3D%22path-5-inside-1_1_194%22%20fill%3D%22white%22%3E%3Cpath%20fill-rule%3D%22evenodd%22%20clip-rule%3D%22evenodd%22%20d%3D%22M747.604%20396.425C747.975%20396.329%20747.975%20395.802%20747.604%20395.706L747.163%20395.593L747.604%20395.724C747.952%20395.827%20747.952%20396.321%20747.604%20396.425ZM488.48%20459.238C488.48%20461.52%20486.936%20463.512%20484.727%20464.081L367.964%20494.132C364.8%20494.946%20361.717%20492.557%20361.717%20489.289L361.717%20302.841C361.717%20299.574%20364.8%20297.185%20367.964%20297.999L484.727%20328.05C486.936%20328.619%20488.48%20330.611%20488.48%20332.892L488.48%20459.238Z%22%2F%3E%3C%2Fmask%3E%3Cpath%20d%3D%22M747.163%20395.593L746.308%20398.468L747.911%20392.687L747.163%20395.593ZM484.727%20328.05L485.474%20325.145L484.727%20328.05ZM488.48%20332.892H485.48H488.48ZM367.964%20297.999L367.216%20300.904L367.964%20297.999ZM488.48%20459.238H491.48H488.48ZM747.604%20395.724L746.748%20398.599L747.604%20395.724ZM747.604%20395.706L748.352%20392.801L747.604%20395.706ZM747.911%20392.687L748.352%20392.801L746.856%20398.611L746.416%20398.498L747.911%20392.687ZM748.019%20392.717L748.46%20392.848L746.748%20398.599L746.308%20398.468L748.019%20392.717ZM485.474%20466.986L368.711%20497.037L367.216%20491.226L483.979%20461.175L485.474%20466.986ZM358.717%20489.289L358.717%20302.841H364.717L364.717%20489.289H358.717ZM368.711%20295.094L485.474%20325.145L483.979%20330.955L367.216%20300.904L368.711%20295.094ZM485.48%20459.238L485.48%20332.892H491.48V459.238L485.48%20459.238ZM485.474%20325.145C489.009%20326.054%20491.48%20329.242%20491.48%20332.892H485.48C485.48%20331.98%20484.863%20331.183%20483.979%20330.955L485.474%20325.145ZM358.717%20302.841C358.717%20297.614%20363.649%20293.791%20368.711%20295.094L367.216%20300.904C365.95%20300.579%20364.717%20301.534%20364.717%20302.841H358.717ZM483.979%20461.175C484.863%20460.948%20485.48%20460.151%20485.48%20459.238L491.48%20459.238C491.48%20462.889%20489.009%20466.076%20485.474%20466.986L483.979%20461.175ZM748.46%20392.848C751.667%20393.803%20751.667%20398.345%20748.46%20399.3L746.748%20393.549C744.238%20394.296%20744.238%20397.852%20746.748%20398.599L748.46%20392.848ZM368.711%20497.037C363.649%20498.34%20358.717%20494.517%20358.717%20489.289H364.717C364.717%20490.596%20365.95%20491.552%20367.216%20491.226L368.711%20497.037ZM746.856%20393.519C744.225%20394.197%20744.225%20397.934%20746.856%20398.611L748.352%20392.801C751.726%20393.669%20751.726%20398.461%20748.352%20399.33L746.856%20393.519Z%22%20fill%3D%22%23B8860B%22%20mask%3D%22url%28%23path-5-inside-1_1_194%29%22%2F%3E%3Cmask%20id%3D%22path-7-inside-2_1_194%22%20fill%3D%22white%22%3E%3Cpath%20fill-rule%3D%22evenodd%22%20clip-rule%3D%22evenodd%22%20d%3D%22M330.401%20395.704C330.029%20395.8%20330.029%20396.33%20330.401%20396.426L330.831%20396.536L330.401%20396.408C330.051%20396.304%20330.051%20395.809%20330.401%20395.704ZM589.52%20332.892C589.52%20330.61%20591.064%20328.618%20593.273%20328.05L710.036%20297.999C713.2%20297.184%20716.283%20299.574%20716.283%20302.841L716.283%20489.289C716.283%20492.556%20713.2%20494.946%20710.036%20494.131L593.273%20464.08C591.064%20463.512%20589.52%20461.52%20589.52%20459.238L589.52%20332.892Z%22%2F%3E%3C%2Fmask%3E%3Cpath%20d%3D%22M330.831%20396.536L331.687%20393.661L330.084%20399.442L330.831%20396.536ZM593.273%20464.08L592.526%20466.986L593.273%20464.08ZM589.52%20459.238L592.52%20459.238L589.52%20459.238ZM710.036%20494.131L710.784%20491.226L710.036%20494.131ZM593.273%20328.05L594.021%20330.955L593.273%20328.05ZM589.52%20332.892L586.52%20332.892L589.52%20332.892ZM330.401%20396.408L329.546%20399.284L330.401%20396.408ZM710.036%20297.999L709.289%20295.093L710.036%20297.999ZM330.401%20396.426L329.654%20399.331L330.401%20396.426ZM330.084%20399.442L329.654%20399.331L331.149%20393.52L331.579%20393.631L330.084%20399.442ZM329.976%20399.412L329.546%20399.284L331.257%20393.533L331.687%20393.661L329.976%20399.412ZM592.526%20325.144L709.289%20295.093L710.784%20300.904L594.021%20330.955L592.526%20325.144ZM719.283%20302.841L719.283%20489.289L713.283%20489.289L713.283%20302.841L719.283%20302.841ZM709.289%20497.037L592.526%20466.986L594.021%20461.175L710.784%20491.226L709.289%20497.037ZM592.52%20332.892L592.52%20459.238L586.52%20459.238L586.52%20332.892L592.52%20332.892ZM592.526%20466.986C588.991%20466.076%20586.52%20462.889%20586.52%20459.238L592.52%20459.238C592.52%20460.151%20593.137%20460.948%20594.021%20461.175L592.526%20466.986ZM719.283%20489.289C719.283%20494.517%20714.351%20498.34%20709.289%20497.037L710.784%20491.226C712.05%20491.552%20713.283%20490.596%20713.283%20489.289L719.283%20489.289ZM594.021%20330.955C593.137%20331.182%20592.52%20331.979%20592.52%20332.892L586.52%20332.892C586.52%20329.242%20588.991%20326.054%20592.526%20325.144L594.021%20330.955ZM329.546%20399.284C326.337%20398.329%20326.337%20393.784%20329.546%20392.829L331.257%20398.58C333.766%20397.833%20333.766%20394.28%20331.257%20393.533L329.546%20399.284ZM709.289%20295.093C714.351%20293.791%20719.283%20297.614%20719.283%20302.841L713.283%20302.841C713.283%20301.534%20712.05%20300.578%20710.784%20300.904L709.289%20295.093ZM331.149%20398.61C333.779%20397.933%20333.779%20394.197%20331.149%20393.52L329.654%20399.331C326.278%20398.462%20326.278%20393.668%20329.654%20392.799L331.149%20398.61Z%22%20fill%3D%22%23B8860B%22%20mask%3D%22url%28%23path-7-inside-2_1_194%29%22%2F%3E%3Cpath%20d%3D%22M533.267%20386.182L538.495%20395.102L543.722%20386.182H547.586L540.54%20397.091L547.586%20408H543.722L538.495%20399.534L533.267%20408H529.404L536.336%20397.091L529.404%20386.182H533.267Z%22%20fill%3D%22%23B8860B%22%2F%3E%3Cpath%20d%3D%22M29.3292%20287V273.909H31.3406V287H29.3292ZM30.3519%20271.727C29.9599%20271.727%2029.6218%20271.594%2029.3377%20271.327C29.0593%20271.06%2028.9201%20270.739%2028.9201%20270.364C28.9201%20269.989%2029.0593%20269.668%2029.3377%20269.401C29.6218%20269.134%2029.9599%20269%2030.3519%20269C30.744%20269%2031.0792%20269.134%2031.3576%20269.401C31.6417%20269.668%2031.7837%20269.989%2031.7837%20270.364C31.7837%20270.739%2031.6417%20271.06%2031.3576%20271.327C31.0792%20271.594%2030.744%20271.727%2030.3519%20271.727ZM37.0359%20279.125V287H35.0245V273.909H36.9677V275.955H37.1381C37.445%20275.29%2037.9109%20274.756%2038.5359%20274.352C39.1609%20273.943%2039.9677%20273.739%2040.9563%20273.739C41.8427%20273.739%2042.6183%20273.92%2043.283%20274.284C43.9478%20274.642%2044.4648%20275.188%2044.8342%20275.92C45.2035%20276.648%2045.3881%20277.568%2045.3881%20278.682V287H43.3768V278.818C43.3768%20277.79%2043.1097%20276.989%2042.5756%20276.415C42.0415%20275.835%2041.3086%20275.545%2040.3768%20275.545C39.7347%20275.545%2039.1609%20275.685%2038.6552%20275.963C38.1552%20276.241%2037.7603%20276.648%2037.4705%20277.182C37.1808%20277.716%2037.0359%20278.364%2037.0359%20279.125ZM49.0636%20291.909V273.909H51.0067V275.989H51.2454C51.3931%20275.761%2051.5977%20275.472%2051.859%20275.119C52.1261%20274.761%2052.5067%20274.443%2053.0011%20274.165C53.5011%20273.881%2054.1772%20273.739%2055.0295%20273.739C56.1317%20273.739%2057.1033%20274.014%2057.9442%20274.565C58.7852%20275.116%2059.4414%20275.898%2059.913%20276.909C60.3846%20277.92%2060.6204%20279.114%2060.6204%20280.489C60.6204%20281.875%2060.3846%20283.077%2059.913%20284.094C59.4414%20285.105%2058.788%20285.889%2057.9528%20286.446C57.1175%20286.997%2056.1545%20287.273%2055.0636%20287.273C54.2227%20287.273%2053.5494%20287.134%2053.0437%20286.855C52.538%20286.571%2052.1488%20286.25%2051.8761%20285.892C51.6033%20285.528%2051.3931%20285.227%2051.2454%20284.989H51.0749V291.909H49.0636ZM51.0408%20280.455C51.0408%20281.443%2051.1857%20282.315%2051.4755%20283.071C51.7653%20283.821%2052.1886%20284.409%2052.7454%20284.835C53.3022%20285.256%2053.984%20285.466%2054.7908%20285.466C55.6317%20285.466%2056.3335%20285.244%2056.896%20284.801C57.4641%20284.352%2057.8903%20283.75%2058.1744%20282.994C58.4641%20282.233%2058.609%20281.386%2058.609%20280.455C58.609%20279.534%2058.467%20278.705%2058.1829%20277.966C57.9045%20277.222%2057.4812%20276.634%2056.913%20276.202C56.3505%20275.764%2055.6431%20275.545%2054.7908%20275.545C53.9727%20275.545%2053.2852%20275.753%2052.7283%20276.168C52.1715%20276.577%2051.7511%20277.151%2051.467%20277.889C51.1829%20278.622%2051.0408%20279.477%2051.0408%20280.455ZM71.9386%20281.648V273.909H73.9499V287H71.9386V284.784H71.8022C71.4954%20285.449%2071.0181%20286.014%2070.3704%20286.48C69.7227%20286.94%2068.9045%20287.17%2067.9158%20287.17C67.0977%20287.17%2066.3704%20286.991%2065.734%20286.634C65.0977%20286.27%2064.5977%20285.724%2064.234%20284.997C63.8704%20284.264%2063.6886%20283.341%2063.6886%20282.227V273.909H65.6999V282.091C65.6999%20283.045%2065.967%20283.807%2066.5011%20284.375C67.0408%20284.943%2067.7283%20285.227%2068.5636%20285.227C69.0636%20285.227%2069.5721%20285.099%2070.0891%20284.844C70.6119%20284.588%2071.0494%20284.196%2071.4016%20283.668C71.7596%20283.139%2071.9386%20282.466%2071.9386%20281.648ZM83.3271%20273.909V275.614H76.543V273.909H83.3271ZM78.5202%20270.773H80.5316V283.25C80.5316%20283.818%2080.614%20284.244%2080.7788%20284.528C80.9492%20284.807%2081.1651%20284.994%2081.4265%20285.091C81.6935%20285.182%2081.9748%20285.227%2082.2702%20285.227C82.4918%20285.227%2082.6737%20285.216%2082.8157%20285.193C82.9577%20285.165%2083.0714%20285.142%2083.1566%20285.125L83.5657%20286.932C83.4293%20286.983%2083.239%20287.034%2082.9947%20287.085C82.7504%20287.142%2082.4407%20287.17%2082.0657%20287.17C81.4975%20287.17%2080.9407%20287.048%2080.3952%20286.804C79.8555%20286.56%2079.4066%20286.188%2079.0487%20285.688C78.6964%20285.188%2078.5202%20284.557%2078.5202%20283.795V270.773Z%22%20fill%3D%22%238D8D8D%22%2F%3E%3Cpath%20d%3D%22M1012.49%20221.568H1051.51V224.639H1012.49V221.568ZM1012.49%20231.964H1051.51V235.036H1012.49V231.964ZM1012.49%20242.358H1039.78V245.429H1012.49V242.358ZM1009.03%20212C1007.6%20212%201006.4%20212.473%201005.44%20213.419C1004.48%20214.365%201004%20215.546%201004%20216.963V250.037C1004%20251.451%201004.48%20252.632%201005.44%20253.578C1006.4%20254.524%201007.6%20254.998%201009.02%20255H1054.98C1056.41%20255%201057.6%20254.526%201058.56%20253.578C1059.52%20252.63%201060%20251.449%201060%20250.037V216.96C1060%20215.548%201059.52%20214.367%201058.56%20213.419C1057.6%20212.471%201056.4%20211.998%201054.98%20212H1009.03ZM1009.03%20215.071H1054.98C1055.46%20215.071%201055.89%20215.268%201056.29%20215.661C1056.69%20216.054%201056.89%20216.488%201056.89%20216.963V250.037C1056.89%20250.51%201056.69%20250.944%201056.29%20251.339C1055.9%20251.734%201055.46%20251.931%201054.98%20251.929H1009.02C1008.55%20251.929%201008.11%20251.732%201007.71%20251.339C1007.31%20250.946%201007.11%20250.512%201007.11%20250.037V216.96C1007.11%20216.489%201007.31%20216.056%201007.71%20215.661C1008.11%20215.266%201008.55%20215.069%201009.02%20215.071%22%20fill%3D%22%238D8D8D%22%2F%3E%3Cpath%20d%3D%22M1009.03%20215.071H1054.98C1055.46%20215.071%201055.89%20215.268%201056.29%20215.661C1056.69%20216.054%201056.89%20216.488%201056.89%20216.963V250.037C1056.89%20250.51%201056.69%20250.944%201056.29%20251.339C1055.9%20251.734%201055.46%20251.931%201054.98%20251.929H1009.02C1008.55%20251.929%201008.11%20251.732%201007.71%20251.339C1007.31%20250.946%201007.11%20250.512%201007.11%20250.037V216.96C1007.11%20216.489%201007.31%20216.056%201007.71%20215.661C1008.11%20215.266%201008.55%20215.069%201009.02%20215.071M1007.11%20215.071V251.929M1012.49%20221.568H1051.51V224.639H1012.49V221.568ZM1012.49%20231.964H1051.51V235.036H1012.49V231.964ZM1012.49%20242.358H1039.78V245.429H1012.49V242.358ZM1009.03%20212C1007.6%20212%201006.4%20212.473%201005.44%20213.419C1004.48%20214.365%201004%20215.546%201004%20216.963V250.037C1004%20251.451%201004.48%20252.632%201005.44%20253.578C1006.4%20254.524%201007.6%20254.998%201009.02%20255H1054.98C1056.41%20255%201057.6%20254.526%201058.56%20253.578C1059.52%20252.63%201060%20251.449%201060%20250.037V216.96C1060%20215.548%201059.52%20214.367%201058.56%20213.419C1057.6%20212.471%201056.4%20211.998%201054.98%20212H1009.03Z%22%20stroke%3D%22%23F4F4F7%22%2F%3E%3Cpath%20d%3D%22M1003.02%20287.273C1001.84%20287.273%201000.8%20286.991%20999.912%20286.429C999.025%20285.866%20998.332%20285.08%20997.832%20284.068C997.338%20283.057%20997.091%20281.875%20997.091%20280.523C997.091%20279.159%20997.338%20277.969%20997.832%20276.952C998.332%20275.935%20999.025%20275.145%20999.912%20274.582C1000.8%20274.02%201001.84%20273.739%201003.02%20273.739C1004.2%20273.739%201005.24%20274.02%201006.12%20274.582C1007.02%20275.145%201007.71%20275.935%201008.2%20276.952C1008.7%20277.969%201008.95%20279.159%201008.95%20280.523C1008.95%20281.875%201008.7%20283.057%201008.2%20284.068C1007.71%20285.08%201007.02%20285.866%201006.12%20286.429C1005.24%20286.991%201004.2%20287.273%201003.02%20287.273ZM1003.02%20285.466C1003.92%20285.466%201004.66%20285.236%201005.24%20284.776C1005.82%20284.315%201006.25%20283.71%201006.53%20282.96C1006.8%20282.21%201006.94%20281.398%201006.94%20280.523C1006.94%20279.648%201006.8%20278.832%201006.53%20278.077C1006.25%20277.321%201005.82%20276.71%201005.24%20276.244C1004.66%20275.778%201003.92%20275.545%201003.02%20275.545C1002.12%20275.545%201001.39%20275.778%201000.81%20276.244C1000.23%20276.71%20999.798%20277.321%20999.52%20278.077C999.241%20278.832%20999.102%20279.648%20999.102%20280.523C999.102%20281.398%20999.241%20282.21%20999.52%20282.96C999.798%20283.71%201000.23%20284.315%201000.81%20284.776C1001.39%20285.236%201002.12%20285.466%201003.02%20285.466ZM1020.27%20281.648V273.909H1022.29V287H1020.27V284.784H1020.14C1019.83%20285.449%201019.35%20286.014%201018.71%20286.48C1018.06%20286.94%201017.24%20287.17%201016.25%20287.17C1015.43%20287.17%201014.71%20286.991%201014.07%20286.634C1013.43%20286.27%201012.93%20285.724%201012.57%20284.997C1012.21%20284.264%201012.02%20283.341%201012.02%20282.227V273.909H1014.04V282.091C1014.04%20283.045%201014.3%20283.807%201014.84%20284.375C1015.38%20284.943%201016.06%20285.227%201016.9%20285.227C1017.4%20285.227%201017.91%20285.099%201018.43%20284.844C1018.95%20284.588%201019.39%20284.196%201019.74%20283.668C1020.1%20283.139%201020.27%20282.466%201020.27%20281.648ZM1031.66%20273.909V275.614H1024.88V273.909H1031.66ZM1026.86%20270.773H1028.87V283.25C1028.87%20283.818%201028.95%20284.244%201029.11%20284.528C1029.29%20284.807%201029.5%20284.994%201029.76%20285.091C1030.03%20285.182%201030.31%20285.227%201030.61%20285.227C1030.83%20285.227%201031.01%20285.216%201031.15%20285.193C1031.29%20285.165%201031.41%20285.142%201031.49%20285.125L1031.9%20286.932C1031.77%20286.983%201031.57%20287.034%201031.33%20287.085C1031.09%20287.142%201030.78%20287.17%201030.4%20287.17C1029.83%20287.17%201029.28%20287.048%201028.73%20286.804C1028.19%20286.56%201027.74%20286.188%201027.38%20285.688C1027.03%20285.188%201026.86%20284.557%201026.86%20283.795V270.773ZM1034.69%20291.909V273.909H1036.63V275.989H1036.87C1037.02%20275.761%201037.22%20275.472%201037.48%20275.119C1037.75%20274.761%201038.13%20274.443%201038.63%20274.165C1039.13%20273.881%201039.8%20273.739%201040.65%20273.739C1041.76%20273.739%201042.73%20274.014%201043.57%20274.565C1044.41%20275.116%201045.07%20275.898%201045.54%20276.909C1046.01%20277.92%201046.25%20279.114%201046.25%20280.489C1046.25%20281.875%201046.01%20283.077%201045.54%20284.094C1045.07%20285.105%201044.41%20285.889%201043.58%20286.446C1042.74%20286.997%201041.78%20287.273%201040.69%20287.273C1039.85%20287.273%201039.17%20287.134%201038.67%20286.855C1038.16%20286.571%201037.77%20286.25%201037.5%20285.892C1037.23%20285.528%201037.02%20285.227%201036.87%20284.989H1036.7V291.909H1034.69ZM1036.67%20280.455C1036.67%20281.443%201036.81%20282.315%201037.1%20283.071C1037.39%20283.821%201037.81%20284.409%201038.37%20284.835C1038.93%20285.256%201039.61%20285.466%201040.42%20285.466C1041.26%20285.466%201041.96%20285.244%201042.52%20284.801C1043.09%20284.352%201043.52%20283.75%201043.8%20282.994C1044.09%20282.233%201044.23%20281.386%201044.23%20280.455C1044.23%20279.534%201044.09%20278.705%201043.81%20277.966C1043.53%20277.222%201043.11%20276.634%201042.54%20276.202C1041.98%20275.764%201041.27%20275.545%201040.42%20275.545C1039.6%20275.545%201038.91%20275.753%201038.35%20276.168C1037.8%20276.577%201037.38%20277.151%201037.09%20277.889C1036.81%20278.622%201036.67%20279.477%201036.67%20280.455ZM1057.56%20281.648V273.909H1059.57V287H1057.56V284.784H1057.43C1057.12%20285.449%201056.64%20286.014%201056%20286.48C1055.35%20286.94%201054.53%20287.17%201053.54%20287.17C1052.72%20287.17%201052%20286.991%201051.36%20286.634C1050.72%20286.27%201050.22%20285.724%201049.86%20284.997C1049.5%20284.264%201049.31%20283.341%201049.31%20282.227V273.909H1051.32V282.091C1051.32%20283.045%201051.59%20283.807%201052.13%20284.375C1052.67%20284.943%201053.35%20285.227%201054.19%20285.227C1054.69%20285.227%201055.2%20285.099%201055.71%20284.844C1056.24%20284.588%201056.67%20284.196%201057.03%20283.668C1057.38%20283.139%201057.56%20282.466%201057.56%20281.648ZM1068.95%20273.909V275.614H1062.17V273.909H1068.95ZM1064.15%20270.773H1066.16V283.25C1066.16%20283.818%201066.24%20284.244%201066.4%20284.528C1066.57%20284.807%201066.79%20284.994%201067.05%20285.091C1067.32%20285.182%201067.6%20285.227%201067.9%20285.227C1068.12%20285.227%201068.3%20285.216%201068.44%20285.193C1068.58%20285.165%201068.7%20285.142%201068.78%20285.125L1069.19%20286.932C1069.05%20286.983%201068.86%20287.034%201068.62%20287.085C1068.38%20287.142%201068.07%20287.17%201067.69%20287.17C1067.12%20287.17%201066.57%20287.048%201066.02%20286.804C1065.48%20286.56%201065.03%20286.188%201064.67%20285.688C1064.32%20285.188%201064.15%20284.557%201064.15%20283.795V270.773Z%22%20fill%3D%22%238D8D8D%22%2F%3E%3Cpath%20d%3D%22M435.207%2079V61.5455H441.105C442.474%2061.5455%20443.594%2061.7926%20444.463%2062.2869C445.338%2062.7756%20445.986%2063.4375%20446.406%2064.2727C446.827%2065.108%20447.037%2066.0398%20447.037%2067.0682C447.037%2068.0966%20446.827%2069.0312%20446.406%2069.8722C445.991%2070.7131%20445.349%2071.3835%20444.48%2071.8835C443.611%2072.3778%20442.497%2072.625%20441.139%2072.625H436.912V70.75H441.071C442.009%2070.75%20442.761%2070.5881%20443.33%2070.2642C443.898%2069.9403%20444.31%2069.5028%20444.565%2068.9517C444.827%2068.3949%20444.957%2067.767%20444.957%2067.0682C444.957%2066.3693%20444.827%2065.7443%20444.565%2065.1932C444.31%2064.642%20443.895%2064.2102%20443.321%2063.8977C442.747%2063.5795%20441.986%2063.4205%20441.037%2063.4205H437.321V79H435.207ZM450.169%2079V65.9091H452.112V67.8864H452.249C452.487%2067.2386%20452.919%2066.7131%20453.544%2066.3097C454.169%2065.9062%20454.874%2065.7045%20455.658%2065.7045C455.805%2065.7045%20455.99%2065.7074%20456.212%2065.7131C456.433%2065.7187%20456.601%2065.7273%20456.714%2065.7386V67.7841C456.646%2067.767%20456.49%2067.7415%20456.246%2067.7074C456.007%2067.6676%20455.754%2067.6477%20455.487%2067.6477C454.851%2067.6477%20454.283%2067.7812%20453.783%2068.0483C453.288%2068.3097%20452.896%2068.6733%20452.607%2069.1392C452.322%2069.5994%20452.18%2070.125%20452.18%2070.7159V79H450.169ZM464.189%2079.2727C462.928%2079.2727%20461.839%2078.9943%20460.925%2078.4375C460.016%2077.875%20459.314%2077.0909%20458.82%2076.0852C458.331%2075.0739%20458.087%2073.8977%20458.087%2072.5568C458.087%2071.2159%20458.331%2070.0341%20458.82%2069.0114C459.314%2067.983%20460.001%2067.1818%20460.882%2066.608C461.768%2066.0284%20462.803%2065.7386%20463.984%2065.7386C464.666%2065.7386%20465.339%2065.8523%20466.004%2066.0795C466.669%2066.3068%20467.274%2066.6761%20467.82%2067.1875C468.365%2067.6932%20468.8%2068.3636%20469.124%2069.1989C469.447%2070.0341%20469.609%2071.0625%20469.609%2072.2841V73.1364H459.518V71.3977H467.564C467.564%2070.6591%20467.416%2070%20467.121%2069.4205C466.831%2068.8409%20466.416%2068.3835%20465.876%2068.0483C465.342%2067.7131%20464.712%2067.5455%20463.984%2067.5455C463.183%2067.5455%20462.49%2067.7443%20461.905%2068.142C461.325%2068.5341%20460.879%2069.0455%20460.567%2069.6761C460.254%2070.3068%20460.098%2070.983%20460.098%2071.7045V72.8636C460.098%2073.8523%20460.268%2074.6903%20460.609%2075.3778C460.956%2076.0597%20461.436%2076.5795%20462.05%2076.9375C462.663%2077.2898%20463.376%2077.4659%20464.189%2077.4659C464.717%2077.4659%20465.195%2077.392%20465.621%2077.2443C466.053%2077.0909%20466.425%2076.8636%20466.737%2076.5625C467.05%2076.2557%20467.291%2075.875%20467.462%2075.4205L469.405%2075.9659C469.2%2076.625%20468.857%2077.2045%20468.374%2077.7045C467.891%2078.1989%20467.294%2078.5852%20466.584%2078.8636C465.874%2079.1364%20465.075%2079.2727%20464.189%2079.2727ZM478.362%2065.9091V67.6136H471.578V65.9091H478.362ZM473.555%2062.7727H475.567V75.25C475.567%2075.8182%20475.649%2076.2443%20475.814%2076.5284C475.984%2076.8068%20476.2%2076.9943%20476.462%2077.0909C476.729%2077.1818%20477.01%2077.2273%20477.305%2077.2273C477.527%2077.2273%20477.709%2077.2159%20477.851%2077.1932C477.993%2077.1648%20478.107%2077.142%20478.192%2077.125L478.601%2078.9318C478.464%2078.983%20478.274%2079.0341%20478.03%2079.0852C477.786%2079.142%20477.476%2079.1705%20477.101%2079.1705C476.533%2079.1705%20475.976%2079.0483%20475.43%2078.804C474.891%2078.5597%20474.442%2078.1875%20474.084%2077.6875C473.732%2077.1875%20473.555%2076.5568%20473.555%2075.7955V62.7727ZM481.388%2079V65.9091H483.331V67.8864H483.467C483.706%2067.2386%20484.138%2066.7131%20484.763%2066.3097C485.388%2065.9062%20486.092%2065.7045%20486.876%2065.7045C487.024%2065.7045%20487.209%2065.7074%20487.43%2065.7131C487.652%2065.7187%20487.82%2065.7273%20487.933%2065.7386V67.7841C487.865%2067.767%20487.709%2067.7415%20487.464%2067.7074C487.226%2067.6676%20486.973%2067.6477%20486.706%2067.6477C486.07%2067.6477%20485.501%2067.7812%20485.001%2068.0483C484.507%2068.3097%20484.115%2068.6733%20483.825%2069.1392C483.541%2069.5994%20483.399%2070.125%20483.399%2070.7159V79H481.388ZM494.17%2079.3068C493.34%2079.3068%20492.587%2079.1506%20491.911%2078.8381C491.235%2078.5199%20490.698%2078.0625%20490.3%2077.4659C489.903%2076.8636%20489.704%2076.1364%20489.704%2075.2841C489.704%2074.5341%20489.852%2073.9261%20490.147%2073.4602C490.442%2072.9886%20490.837%2072.6193%20491.332%2072.3523C491.826%2072.0852%20492.371%2071.8864%20492.968%2071.7557C493.57%2071.6193%20494.175%2071.5114%20494.783%2071.4318C495.579%2071.3295%20496.224%2071.2528%20496.718%2071.2017C497.218%2071.1449%20497.582%2071.0511%20497.809%2070.9205C498.042%2070.7898%20498.158%2070.5625%20498.158%2070.2386V70.1705C498.158%2069.3295%20497.928%2068.6761%20497.468%2068.2102C497.013%2067.7443%20496.323%2067.5114%20495.397%2067.5114C494.437%2067.5114%20493.684%2067.7216%20493.138%2068.142C492.593%2068.5625%20492.21%2069.0114%20491.988%2069.4886L490.079%2068.8068C490.42%2068.0114%20490.874%2067.392%20491.442%2066.9489C492.016%2066.5%20492.641%2066.1875%20493.317%2066.0114C493.999%2065.8295%20494.67%2065.7386%20495.329%2065.7386C495.749%2065.7386%20496.232%2065.7898%20496.778%2065.892C497.329%2065.9886%20497.86%2066.1903%20498.371%2066.4972C498.888%2066.804%20499.317%2067.267%20499.658%2067.8864C499.999%2068.5057%20500.17%2069.3352%20500.17%2070.375V79H498.158V77.2273H498.056C497.92%2077.5114%20497.692%2077.8153%20497.374%2078.1392C497.056%2078.4631%20496.633%2078.7386%20496.104%2078.9659C495.576%2079.1932%20494.931%2079.3068%20494.17%2079.3068ZM494.477%2077.5C495.272%2077.5%20495.942%2077.3438%20496.488%2077.0312C497.039%2076.7188%20497.454%2076.3153%20497.732%2075.821C498.016%2075.3267%20498.158%2074.8068%20498.158%2074.2614V72.4205C498.073%2072.5227%20497.886%2072.6165%20497.596%2072.7017C497.312%2072.7812%20496.982%2072.8523%20496.607%2072.9148C496.238%2072.9716%20495.877%2073.0227%20495.525%2073.0682C495.178%2073.108%20494.897%2073.142%20494.681%2073.1705C494.158%2073.2386%20493.67%2073.3494%20493.215%2073.5028C492.766%2073.6506%20492.403%2073.875%20492.124%2074.1761C491.852%2074.4716%20491.715%2074.875%20491.715%2075.3864C491.715%2076.0852%20491.974%2076.6136%20492.491%2076.9716C493.013%2077.3239%20493.675%2077.5%20494.477%2077.5ZM503.841%2079V65.9091H505.852V79H503.841ZM504.864%2063.7273C504.472%2063.7273%20504.134%2063.5938%20503.849%2063.3267C503.571%2063.0597%20503.432%2062.7386%20503.432%2062.3636C503.432%2061.9886%20503.571%2061.6676%20503.849%2061.4006C504.134%2061.1335%20504.472%2061%20504.864%2061C505.256%2061%20505.591%2061.1335%20505.869%2061.4006C506.153%2061.6676%20506.295%2061.9886%20506.295%2062.3636C506.295%2062.7386%20506.153%2063.0597%20505.869%2063.3267C505.591%2063.5938%20505.256%2063.7273%20504.864%2063.7273ZM511.548%2071.125V79H509.536V65.9091H511.479V67.9545H511.65C511.957%2067.2898%20512.423%2066.7557%20513.048%2066.3523C513.673%2065.9432%20514.479%2065.7386%20515.468%2065.7386C516.354%2065.7386%20517.13%2065.9205%20517.795%2066.2841C518.46%2066.642%20518.977%2067.1875%20519.346%2067.9205C519.715%2068.6477%20519.9%2069.5682%20519.9%2070.6818V79H517.888V70.8182C517.888%2069.7898%20517.621%2068.9886%20517.087%2068.4148C516.553%2067.8352%20515.82%2067.5455%20514.888%2067.5455C514.246%2067.5455%20513.673%2067.6847%20513.167%2067.9631C512.667%2068.2415%20512.272%2068.6477%20511.982%2069.1818C511.692%2069.7159%20511.548%2070.3636%20511.548%2071.125ZM529.064%2079.2727C527.803%2079.2727%20526.714%2078.9943%20525.8%2078.4375C524.891%2077.875%20524.189%2077.0909%20523.695%2076.0852C523.206%2075.0739%20522.962%2073.8977%20522.962%2072.5568C522.962%2071.2159%20523.206%2070.0341%20523.695%2069.0114C524.189%2067.983%20524.876%2067.1818%20525.757%2066.608C526.643%2066.0284%20527.678%2065.7386%20528.859%2065.7386C529.541%2065.7386%20530.214%2065.8523%20530.879%2066.0795C531.544%2066.3068%20532.149%2066.6761%20532.695%2067.1875C533.24%2067.6932%20533.675%2068.3636%20533.999%2069.1989C534.322%2070.0341%20534.484%2071.0625%20534.484%2072.2841V73.1364H524.393V71.3977H532.439C532.439%2070.6591%20532.291%2070%20531.996%2069.4205C531.706%2068.8409%20531.291%2068.3835%20530.751%2068.0483C530.217%2067.7131%20529.587%2067.5455%20528.859%2067.5455C528.058%2067.5455%20527.365%2067.7443%20526.78%2068.142C526.2%2068.5341%20525.754%2069.0455%20525.442%2069.6761C525.129%2070.3068%20524.973%2070.983%20524.973%2071.7045V72.8636C524.973%2073.8523%20525.143%2074.6903%20525.484%2075.3778C525.831%2076.0597%20526.311%2076.5795%20526.925%2076.9375C527.538%2077.2898%20528.251%2077.4659%20529.064%2077.4659C529.592%2077.4659%20530.07%2077.392%20530.496%2077.2443C530.928%2077.0909%20531.3%2076.8636%20531.612%2076.5625C531.925%2076.2557%20532.166%2075.875%20532.337%2075.4205L534.28%2075.9659C534.075%2076.625%20533.732%2077.2045%20533.249%2077.7045C532.766%2078.1989%20532.169%2078.5852%20531.459%2078.8636C530.749%2079.1364%20529.95%2079.2727%20529.064%2079.2727ZM542.487%2079.2727C541.396%2079.2727%20540.433%2078.9972%20539.598%2078.446C538.763%2077.8892%20538.109%2077.1051%20537.638%2076.0938C537.166%2075.0767%20536.93%2073.875%20536.93%2072.4886C536.93%2071.1136%20537.166%2069.9205%20537.638%2068.9091C538.109%2067.8977%20538.766%2067.1165%20539.607%2066.5653C540.447%2066.0142%20541.419%2065.7386%20542.521%2065.7386C543.374%2065.7386%20544.047%2065.8807%20544.541%2066.1648C545.041%2066.4432%20545.422%2066.7614%20545.683%2067.1193C545.95%2067.4716%20546.158%2067.7614%20546.305%2067.9886H546.476V61.5455H548.487V79H546.544V76.9886H546.305C546.158%2077.2273%20545.947%2077.5284%20545.675%2077.892C545.402%2078.25%20545.013%2078.571%20544.507%2078.8551C544.001%2079.1335%20543.328%2079.2727%20542.487%2079.2727ZM542.76%2077.4659C543.567%2077.4659%20544.249%2077.2557%20544.805%2076.8352C545.362%2076.4091%20545.786%2075.821%20546.075%2075.071C546.365%2074.3153%20546.51%2073.4432%20546.51%2072.4545C546.51%2071.4773%20546.368%2070.6222%20546.084%2069.8892C545.8%2069.1506%20545.379%2068.5767%20544.822%2068.1676C544.266%2067.7528%20543.578%2067.5455%20542.76%2067.5455C541.908%2067.5455%20541.197%2067.7642%20540.629%2068.2017C540.067%2068.6335%20539.643%2069.2216%20539.359%2069.9659C539.081%2070.7045%20538.942%2071.5341%20538.942%2072.4545C538.942%2073.3864%20539.084%2074.233%20539.368%2074.9943C539.658%2075.75%20540.084%2076.3523%20540.646%2076.8011C541.214%2077.2443%20541.919%2077.4659%20542.76%2077.4659ZM562.166%2079L558.178%2065.9091H560.291L563.121%2075.9318H563.257L566.053%2065.9091H568.2L570.962%2075.8977H571.098L573.928%2065.9091H576.041L572.053%2079H570.075L567.212%2068.9432H567.007L564.143%2079H562.166ZM583.744%2079.2727C582.482%2079.2727%20581.394%2078.9943%20580.479%2078.4375C579.57%2077.875%20578.869%2077.0909%20578.374%2076.0852C577.886%2075.0739%20577.641%2073.8977%20577.641%2072.5568C577.641%2071.2159%20577.886%2070.0341%20578.374%2069.0114C578.869%2067.983%20579.556%2067.1818%20580.437%2066.608C581.323%2066.0284%20582.357%2065.7386%20583.539%2065.7386C584.221%2065.7386%20584.894%2065.8523%20585.559%2066.0795C586.224%2066.3068%20586.829%2066.6761%20587.374%2067.1875C587.92%2067.6932%20588.354%2068.3636%20588.678%2069.1989C589.002%2070.0341%20589.164%2071.0625%20589.164%2072.2841V73.1364H579.073V71.3977H587.119C587.119%2070.6591%20586.971%2070%20586.675%2069.4205C586.386%2068.8409%20585.971%2068.3835%20585.431%2068.0483C584.897%2067.7131%20584.266%2067.5455%20583.539%2067.5455C582.738%2067.5455%20582.045%2067.7443%20581.46%2068.142C580.88%2068.5341%20580.434%2069.0455%20580.121%2069.6761C579.809%2070.3068%20579.653%2070.983%20579.653%2071.7045V72.8636C579.653%2073.8523%20579.823%2074.6903%20580.164%2075.3778C580.511%2076.0597%20580.991%2076.5795%20581.604%2076.9375C582.218%2077.2898%20582.931%2077.4659%20583.744%2077.4659C584.272%2077.4659%20584.749%2077.392%20585.175%2077.2443C585.607%2077.0909%20585.979%2076.8636%20586.292%2076.5625C586.604%2076.2557%20586.846%2075.875%20587.016%2075.4205L588.96%2075.9659C588.755%2076.625%20588.411%2077.2045%20587.928%2077.7045C587.445%2078.1989%20586.849%2078.5852%20586.138%2078.8636C585.428%2079.1364%20584.63%2079.2727%20583.744%2079.2727ZM592.224%2079V65.9091H594.235V79H592.224ZM593.246%2063.7273C592.854%2063.7273%20592.516%2063.5938%20592.232%2063.3267C591.954%2063.0597%20591.815%2062.7386%20591.815%2062.3636C591.815%2061.9886%20591.954%2061.6676%20592.232%2061.4006C592.516%2061.1335%20592.854%2061%20593.246%2061C593.638%2061%20593.974%2061.1335%20594.252%2061.4006C594.536%2061.6676%20594.678%2061.9886%20594.678%2062.3636C594.678%2062.7386%20594.536%2063.0597%20594.252%2063.3267C593.974%2063.5938%20593.638%2063.7273%20593.246%2063.7273ZM603.203%2084.1818C602.232%2084.1818%20601.396%2084.0568%20600.697%2083.8068C599.999%2083.5625%20599.416%2083.2386%20598.95%2082.8352C598.49%2082.4375%20598.124%2082.0114%20597.851%2081.5568L599.453%2080.4318C599.635%2080.6705%20599.865%2080.9432%20600.143%2081.25C600.422%2081.5625%20600.803%2081.8324%20601.286%2082.0597C601.774%2082.2926%20602.413%2082.4091%20603.203%2082.4091C604.26%2082.4091%20605.132%2082.1534%20605.82%2081.642C606.507%2081.1307%20606.851%2080.3295%20606.851%2079.2386V76.5795H606.68C606.533%2076.8182%20606.322%2077.1136%20606.05%2077.4659C605.783%2077.8125%20605.396%2078.1222%20604.891%2078.3949C604.391%2078.6619%20603.714%2078.7955%20602.862%2078.7955C601.805%2078.7955%20600.857%2078.5455%20600.016%2078.0455C599.18%2077.5455%20598.518%2076.8182%20598.03%2075.8636C597.547%2074.9091%20597.305%2073.75%20597.305%2072.3864C597.305%2071.0455%20597.541%2069.8778%20598.013%2068.8835C598.484%2067.8835%20599.141%2067.1108%20599.982%2066.5653C600.822%2066.0142%20601.794%2065.7386%20602.896%2065.7386C603.749%2065.7386%20604.425%2065.8807%20604.925%2066.1648C605.43%2066.4432%20605.817%2066.7614%20606.084%2067.1193C606.357%2067.4716%20606.567%2067.7614%20606.714%2067.9886H606.919V65.9091H608.862V79.375C608.862%2080.5%20608.607%2081.4148%20608.095%2082.1193C607.589%2082.8295%20606.908%2083.3494%20606.05%2083.679C605.197%2084.0142%20604.249%2084.1818%20603.203%2084.1818ZM603.135%2076.9886C603.942%2076.9886%20604.624%2076.804%20605.18%2076.4347C605.737%2076.0653%20606.161%2075.5341%20606.45%2074.8409C606.74%2074.1477%20606.885%2073.3182%20606.885%2072.3523C606.885%2071.4091%20606.743%2070.5767%20606.459%2069.8551C606.175%2069.1335%20605.754%2068.5682%20605.197%2068.1591C604.641%2067.75%20603.953%2067.5455%20603.135%2067.5455C602.283%2067.5455%20601.572%2067.7614%20601.004%2068.1932C600.442%2068.625%20600.018%2069.2045%20599.734%2069.9318C599.456%2070.6591%20599.317%2071.4659%20599.317%2072.3523C599.317%2073.2614%20599.459%2074.0653%20599.743%2074.7642C600.033%2075.4574%20600.459%2076.0028%20601.021%2076.4006C601.589%2076.7926%20602.294%2076.9886%20603.135%2076.9886ZM614.555%2071.125V79H612.544V61.5455H614.555V67.9545H614.726C615.033%2067.2784%20615.493%2066.7415%20616.107%2066.3438C616.726%2065.9403%20617.55%2065.7386%20618.578%2065.7386C619.47%2065.7386%20620.251%2065.9176%20620.922%2066.2756C621.592%2066.6278%20622.112%2067.1705%20622.482%2067.9034C622.857%2068.6307%20623.044%2069.5568%20623.044%2070.6818V79H621.033V70.8182C621.033%2069.7784%20620.763%2068.9744%20620.223%2068.4062C619.689%2067.8324%20618.947%2067.5455%20617.999%2067.5455C617.339%2067.5455%20616.749%2067.6847%20616.226%2067.9631C615.709%2068.2415%20615.3%2068.6477%20614.999%2069.1818C614.703%2069.7159%20614.555%2070.3636%20614.555%2071.125ZM632.417%2065.9091V67.6136H625.633V65.9091H632.417ZM627.61%2062.7727H629.621V75.25C629.621%2075.8182%20629.704%2076.2443%20629.869%2076.5284C630.039%2076.8068%20630.255%2076.9943%20630.516%2077.0909C630.783%2077.1818%20631.065%2077.2273%20631.36%2077.2273C631.582%2077.2273%20631.763%2077.2159%20631.906%2077.1932C632.048%2077.1648%20632.161%2077.142%20632.246%2077.125L632.656%2078.9318C632.519%2078.983%20632.329%2079.0341%20632.085%2079.0852C631.84%2079.142%20631.531%2079.1705%20631.156%2079.1705C630.587%2079.1705%20630.031%2079.0483%20629.485%2078.804C628.945%2078.5597%20628.496%2078.1875%20628.138%2077.6875C627.786%2077.1875%20627.61%2076.5568%20627.61%2075.7955V62.7727ZM644.481%2068.8409L642.674%2069.3523C642.56%2069.0511%20642.393%2068.7585%20642.171%2068.4744C641.955%2068.1847%20641.66%2067.946%20641.285%2067.7585C640.91%2067.571%20640.43%2067.4773%20639.844%2067.4773C639.043%2067.4773%20638.376%2067.6619%20637.842%2068.0312C637.313%2068.3949%20637.049%2068.858%20637.049%2069.4205C637.049%2069.9205%20637.231%2070.3153%20637.594%2070.6051C637.958%2070.8949%20638.526%2071.1364%20639.299%2071.3295L641.242%2071.8068C642.413%2072.0909%20643.285%2072.5256%20643.859%2073.1108C644.433%2073.6903%20644.719%2074.4375%20644.719%2075.3523C644.719%2076.1023%20644.504%2076.7727%20644.072%2077.3636C643.646%2077.9545%20643.049%2078.4205%20642.282%2078.7614C641.515%2079.1023%20640.623%2079.2727%20639.606%2079.2727C638.271%2079.2727%20637.165%2078.983%20636.29%2078.4034C635.415%2077.8239%20634.862%2076.9773%20634.629%2075.8636L636.538%2075.3864C636.719%2076.0909%20637.063%2076.6193%20637.569%2076.9716C638.08%2077.3239%20638.748%2077.5%20639.572%2077.5C640.509%2077.5%20641.254%2077.3011%20641.805%2076.9034C642.362%2076.5%20642.64%2076.017%20642.64%2075.4545C642.64%2075%20642.481%2074.6193%20642.163%2074.3125C641.844%2074%20641.356%2073.767%20640.697%2073.6136L638.515%2073.1023C637.316%2072.8182%20636.435%2072.3778%20635.873%2071.7812C635.316%2071.179%20635.038%2070.4261%20635.038%2069.5227C635.038%2068.7841%20635.245%2068.1307%20635.66%2067.5625C636.08%2066.9943%20636.651%2066.5483%20637.373%2066.2244C638.1%2065.9006%20638.924%2065.7386%20639.844%2065.7386C641.14%2065.7386%20642.157%2066.0227%20642.896%2066.5909C643.64%2067.1591%20644.168%2067.9091%20644.481%2068.8409ZM497.303%20100.636C497.303%2098.4886%20497.581%2096.5142%20498.138%2094.7131C498.701%2092.9062%20499.502%2091.2443%20500.542%2089.7273H502.314C501.905%2090.2898%20501.522%2090.983%20501.164%2091.8068C500.811%2092.625%20500.502%2093.5256%20500.235%2094.5085C499.968%2095.4858%20499.757%2096.4972%20499.604%2097.5426C499.456%2098.5881%20499.382%2099.6193%20499.382%20100.636C499.382%20101.989%20499.513%20103.361%20499.775%20104.753C500.036%20106.145%20500.388%20107.437%20500.831%20108.631C501.275%20109.824%20501.769%20110.795%20502.314%20111.545H500.542C499.502%20110.028%20498.701%20108.369%20498.138%20106.568C497.581%20104.761%20497.303%20102.784%20497.303%20100.636ZM511.112%2094.9091V96.6136H504.055V94.9091H511.112ZM506.169%20108V93.1023C506.169%2092.3523%20506.345%2091.7273%20506.697%2091.2273C507.049%2090.7273%20507.507%2090.3523%20508.069%2090.1023C508.632%2089.8523%20509.225%2089.7273%20509.85%2089.7273C510.345%2089.7273%20510.748%2089.767%20511.061%2089.8466C511.373%2089.9261%20511.606%2090%20511.76%2090.0682L511.18%2091.8068C511.078%2091.7727%20510.936%2091.7301%20510.754%2091.679C510.578%2091.6278%20510.345%2091.6023%20510.055%2091.6023C509.39%2091.6023%20508.91%2091.7699%20508.615%2092.1051C508.325%2092.4403%20508.18%2092.9318%20508.18%2093.5795V108H506.169ZM513.931%20108V94.9091H515.874V96.8864H516.01C516.249%2096.2386%20516.681%2095.7131%20517.306%2095.3097C517.931%2094.9062%20518.635%2094.7045%20519.419%2094.7045C519.567%2094.7045%20519.752%2094.7074%20519.973%2094.7131C520.195%2094.7187%20520.363%2094.7273%20520.476%2094.7386V96.7841C520.408%2096.767%20520.252%2096.7415%20520.007%2096.7074C519.769%2096.6676%20519.516%2096.6477%20519.249%2096.6477C518.613%2096.6477%20518.044%2096.7812%20517.544%2097.0483C517.05%2097.3097%20516.658%2097.6733%20516.368%2098.1392C516.084%2098.5994%20515.942%2099.125%20515.942%2099.7159V108H513.931ZM527.78%20108.273C526.598%20108.273%20525.561%20107.991%20524.669%20107.429C523.783%20106.866%20523.09%20106.08%20522.59%20105.068C522.096%20104.057%20521.848%20102.875%20521.848%20101.523C521.848%20100.159%20522.096%2098.9687%20522.59%2097.9517C523.09%2096.9347%20523.783%2096.1449%20524.669%2095.5824C525.561%2095.0199%20526.598%2094.7386%20527.78%2094.7386C528.962%2094.7386%20529.996%2095.0199%20530.882%2095.5824C531.775%2096.1449%20532.468%2096.9347%20532.962%2097.9517C533.462%2098.9687%20533.712%20100.159%20533.712%20101.523C533.712%20102.875%20533.462%20104.057%20532.962%20105.068C532.468%20106.08%20531.775%20106.866%20530.882%20107.429C529.996%20107.991%20528.962%20108.273%20527.78%20108.273ZM527.78%20106.466C528.678%20106.466%20529.417%20106.236%20529.996%20105.776C530.576%20105.315%20531.005%20104.71%20531.283%20103.96C531.561%20103.21%20531.701%20102.398%20531.701%20101.523C531.701%20100.648%20531.561%2099.8324%20531.283%2099.0767C531.005%2098.321%20530.576%2097.7102%20529.996%2097.2443C529.417%2096.7784%20528.678%2096.5455%20527.78%2096.5455C526.882%2096.5455%20526.144%2096.7784%20525.564%2097.2443C524.985%2097.7102%20524.556%2098.321%20524.277%2099.0767C523.999%2099.8324%20523.86%20100.648%20523.86%20101.523C523.86%20102.398%20523.999%20103.21%20524.277%20103.96C524.556%20104.71%20524.985%20105.315%20525.564%20105.776C526.144%20106.236%20526.882%20106.466%20527.78%20106.466ZM536.407%20108V106.466L543.839%2096.9205V96.7841H536.646V94.9091H546.43V96.5114L539.203%20105.989V106.125H546.669V108H536.407ZM554.88%20108.273C553.619%20108.273%20552.531%20107.994%20551.616%20107.438C550.707%20106.875%20550.005%20106.091%20549.511%20105.085C549.022%20104.074%20548.778%20102.898%20548.778%20101.557C548.778%20100.216%20549.022%2099.0341%20549.511%2098.0114C550.005%2096.983%20550.693%2096.1818%20551.574%2095.608C552.46%2095.0284%20553.494%2094.7386%20554.676%2094.7386C555.358%2094.7386%20556.031%2094.8523%20556.696%2095.0795C557.36%2095.3068%20557.966%2095.6761%20558.511%2096.1875C559.056%2096.6932%20559.491%2097.3636%20559.815%2098.1989C560.139%2099.0341%20560.301%20100.062%20560.301%20101.284V102.136H550.21V100.398H558.255C558.255%2099.6591%20558.108%2099%20557.812%2098.4205C557.522%2097.8409%20557.108%2097.3835%20556.568%2097.0483C556.034%2096.7131%20555.403%2096.5455%20554.676%2096.5455C553.875%2096.5455%20553.181%2096.7443%20552.596%2097.142C552.017%2097.5341%20551.571%2098.0455%20551.258%2098.6761C550.946%2099.3068%20550.789%2099.983%20550.789%20100.705V101.864C550.789%20102.852%20550.96%20103.69%20551.301%20104.378C551.647%20105.06%20552.127%20105.58%20552.741%20105.938C553.355%20106.29%20554.068%20106.466%20554.88%20106.466C555.409%20106.466%20555.886%20106.392%20556.312%20106.244C556.744%20106.091%20557.116%20105.864%20557.429%20105.562C557.741%20105.256%20557.983%20104.875%20558.153%20104.42L560.096%20104.966C559.892%20105.625%20559.548%20106.205%20559.065%20106.705C558.582%20107.199%20557.985%20107.585%20557.275%20107.864C556.565%20108.136%20555.767%20108.273%20554.88%20108.273ZM565.372%20100.125V108H563.36V94.9091H565.304V96.9545H565.474C565.781%2096.2898%20566.247%2095.7557%20566.872%2095.3523C567.497%2094.9432%20568.304%2094.7386%20569.292%2094.7386C570.179%2094.7386%20570.954%2094.9205%20571.619%2095.2841C572.284%2095.642%20572.801%2096.1875%20573.17%2096.9205C573.539%2097.6477%20573.724%2098.5682%20573.724%2099.6818V108H571.713V99.8182C571.713%2098.7898%20571.446%2097.9886%20570.912%2097.4148C570.377%2096.8352%20569.645%2096.5455%20568.713%2096.5455C568.071%2096.5455%20567.497%2096.6847%20566.991%2096.9631C566.491%2097.2415%20566.096%2097.6477%20565.806%2098.1818C565.517%2098.7159%20565.372%2099.3636%20565.372%20100.125ZM581.695%20100.636C581.695%20102.784%20581.414%20104.761%20580.851%20106.568C580.294%20108.369%20579.496%20110.028%20578.456%20111.545H576.684C577.093%20110.983%20577.473%20110.29%20577.826%20109.466C578.184%20108.648%20578.496%20107.75%20578.763%20106.773C579.03%20105.79%20579.238%20104.776%20579.385%20103.73C579.539%20102.679%20579.615%20101.648%20579.615%20100.636C579.615%2099.2841%20579.485%2097.9119%20579.223%2096.5199C578.962%2095.1278%20578.61%2093.8352%20578.167%2092.642C577.723%2091.4489%20577.229%2090.4773%20576.684%2089.7273H578.456C579.496%2091.2443%20580.294%2092.9062%20580.851%2094.7131C581.414%2096.5142%20581.695%2098.4886%20581.695%20100.636Z%22%20fill%3D%22%238D8D8D%22%2F%3E%3Cpath%20d%3D%22M191.707%20255.707C192.098%20255.317%20192.098%20254.683%20191.707%20254.293L185.343%20247.929C184.953%20247.538%20184.319%20247.538%20183.929%20247.929C183.538%20248.319%20183.538%20248.953%20183.929%20249.343L189.586%20255L183.929%20260.657C183.538%20261.047%20183.538%20261.681%20183.929%20262.071C184.319%20262.462%20184.953%20262.462%20185.343%20262.071L191.707%20255.707ZM112%20256H191V254H112V256Z%22%20fill%3D%22%23D9D9D9%22%2F%3E%3Cpath%20d%3D%22M967.707%20254.707C968.098%20254.317%20968.098%20253.683%20967.707%20253.293L961.343%20246.929C960.953%20246.538%20960.319%20246.538%20959.929%20246.929C959.538%20247.319%20959.538%20247.953%20959.929%20248.343L965.586%20254L959.929%20259.657C959.538%20260.047%20959.538%20260.681%20959.929%20261.071C960.319%20261.462%20960.953%20261.462%20961.343%20261.071L967.707%20254.707ZM888%20255H967V253H888V255Z%22%20fill%3D%22%23D9D9D9%22%2F%3E%3Cpath%20d%3D%22M303.707%20410.293C304.098%20410.683%20304.098%20411.317%20303.707%20411.707L297.343%20418.071C296.953%20418.462%20296.319%20418.462%20295.929%20418.071C295.538%20417.681%20295.538%20417.047%20295.929%20416.657L301.586%20411L295.929%20405.343C295.538%20404.953%20295.538%20404.319%20295.929%20403.929C296.319%20403.538%20296.953%20403.538%20297.343%20403.929L303.707%20410.293ZM285.4%20411V412V411ZM250.2%20376.444H249.2H250.2ZM250.2%20290.056H251.2H250.2ZM215%20256.5C214.448%20256.5%20214%20256.052%20214%20255.5C214%20254.948%20214.448%20254.5%20215%20254.5V256.5ZM250.2%20220.944H251.2H250.2ZM303.707%2099.2929C304.098%2099.6834%20304.098%20100.317%20303.707%20100.707L297.343%20107.071C296.953%20107.462%20296.319%20107.462%20295.929%20107.071C295.538%20106.681%20295.538%20106.047%20295.929%20105.657L301.586%20100L295.929%2094.3431C295.538%2093.9526%20295.538%2093.3195%20295.929%2092.9289C296.319%2092.5384%20296.953%2092.5384%20297.343%2092.9289L303.707%2099.2929ZM303%20412H285.4V410H303V412ZM285.4%20412C275.805%20412%20266.599%20408.258%20259.809%20401.593L261.21%20400.165C267.623%20406.46%20276.324%20410%20285.4%20410V412ZM259.809%20401.593C253.018%20394.926%20249.2%20385.88%20249.2%20376.444H251.2C251.2%20385.338%20254.799%20393.871%20261.21%20400.165L259.809%20401.593ZM249.2%20376.444V290.056H251.2V376.444H249.2ZM249.2%20290.056C249.2%20281.162%20245.601%20272.629%20239.19%20266.335L240.591%20264.907C247.382%20271.574%20251.2%20280.62%20251.2%20290.056H249.2ZM239.19%20266.335C232.777%20260.04%20224.076%20256.5%20215%20256.5V254.5C224.595%20254.5%20233.801%20258.242%20240.591%20264.907L239.19%20266.335ZM215%20254.5C224.076%20254.5%20232.777%20250.96%20239.19%20244.665L240.591%20246.092C233.801%20252.758%20224.595%20256.5%20215%20256.5V254.5ZM239.19%20244.665C245.601%20238.371%20249.2%20229.838%20249.2%20220.944H251.2C251.2%20230.38%20247.382%20239.426%20240.591%20246.092L239.19%20244.665ZM249.2%20220.944V134.556H251.2V220.944H249.2ZM249.2%20134.556C249.2%20114.98%20265.505%2099%20285.4%2099V101C266.575%20101%20251.2%20116.12%20251.2%20134.556H249.2ZM285.4%2099H303V101H285.4V99Z%22%20fill%3D%22%23D9D9D9%22%2F%3E%3Cpath%20d%3D%22M776%2099L793.6%2099C802.936%2099%20811.889%20102.641%20818.49%20109.121C825.091%20115.602%20828.8%20124.391%20828.8%20133.556L828.8%20219.944C828.8%20229.109%20832.509%20237.898%20839.11%20244.379C845.711%20250.859%20854.664%20254.5%20864%20254.5C854.664%20254.5%20845.711%20258.141%20839.11%20264.621C832.509%20271.102%20828.8%20279.891%20828.8%20289.056L828.8%20375.444C828.8%20394.45%20812.96%20410%20793.6%20410L776%20410%22%20stroke%3D%22%23D9D9D9%22%20stroke-width%3D%222%22%20stroke-linecap%3D%22round%22%20stroke-linejoin%3D%22round%22%2F%3E%3Cpath%20d%3D%22M471.119%20157.829C471.461%20157.829%20472.072%20157.841%20472.953%20157.866C473.833%20157.89%20474.652%20157.914%20475.41%20157.939C476.657%20157.939%20477.696%20157.927%20478.527%20157.902C479.358%20157.878%20479.835%20157.866%20479.957%20157.866C480.397%20157.866%20480.617%20157.963%20480.617%20158.159C480.617%20158.257%20480.581%20158.44%20480.507%20158.709C480.385%20159.149%20480.275%20159.394%20480.177%20159.442C480.079%20159.491%20479.81%20159.516%20479.37%20159.516C478.759%20159.54%20478.27%20159.601%20477.903%20159.699C477.537%20159.797%20477.28%20159.895%20477.133%20159.993C476.987%20160.09%20476.95%20160.2%20477.023%20160.323C477.023%20160.616%20477.219%20163.525%20477.61%20169.051C478.001%20174.576%20478.197%20177.498%20478.197%20177.815L478.27%20177.705C478.319%20177.632%20478.392%20177.51%20478.49%20177.339C478.588%20177.168%20478.735%20176.935%20478.93%20176.642C479.126%20176.349%20479.334%20175.982%20479.554%20175.542C479.774%20175.102%20480.067%20174.6%20480.434%20174.038C480.801%20173.476%20481.167%20172.816%20481.534%20172.058C481.901%20171.3%20482.378%20170.481%20482.964%20169.601C485.507%20165.102%20486.949%20162.535%20487.292%20161.9C487.634%20161.264%20487.817%20160.873%20487.842%20160.726C487.842%20160.359%20487.61%20160.066%20487.145%20159.846C486.68%20159.626%20486.167%20159.516%20485.605%20159.516C485.262%20159.516%20485.091%20159.394%20485.091%20159.149C485.091%20159.076%20485.116%20158.917%20485.165%20158.672C485.262%20158.281%20485.336%20158.049%20485.385%20157.976C485.434%20157.902%20485.605%20157.866%20485.898%20157.866C485.971%20157.866%20486.179%20157.866%20486.522%20157.866C486.864%20157.866%20487.341%20157.866%20487.952%20157.866C488.563%20157.866%20489.223%20157.878%20489.932%20157.902C491.546%20157.902%20492.585%20157.878%20493.049%20157.829C493.392%20157.829%20493.563%20157.963%20493.563%20158.232C493.563%20158.379%20493.526%20158.587%20493.453%20158.856C493.379%20159.125%20493.294%20159.308%20493.196%20159.406C493.098%20159.504%20492.976%20159.54%20492.829%20159.516C492.291%20159.516%20491.802%20159.589%20491.362%20159.736C490.922%20159.883%20490.556%20160.054%20490.262%20160.249C489.969%20160.445%20489.737%20160.653%20489.565%20160.873C489.394%20161.093%20489.248%20161.288%20489.125%20161.459L488.979%20161.679L482.781%20172.535C478.674%20179.771%20476.608%20183.402%20476.583%20183.426C476.437%20183.598%20476.143%20183.683%20475.703%20183.683C475.336%20183.683%20475.104%20183.622%20475.006%20183.5C474.933%20183.426%20474.701%20180.419%20474.31%20174.478C473.918%20168.537%20473.711%20165.469%20473.686%20165.273L473.246%20165.934C472.708%20166.838%20471.143%20169.564%20468.552%20174.112C465.031%20180.273%20463.247%20183.377%20463.198%20183.426C463.149%20183.475%20463.088%20183.524%20463.014%20183.573C462.892%20183.646%20462.672%20183.683%20462.354%20183.683C461.89%20183.683%20461.633%20183.598%20461.584%20183.426C461.584%20183.353%20461.315%20179.454%20460.777%20171.728C460.239%20164.002%20459.958%20160.115%20459.934%20160.066C459.885%20159.846%20459.763%20159.711%20459.567%20159.662C459.372%20159.614%20458.895%20159.565%20458.137%20159.516H457.257C457.086%20159.345%20457%20159.21%20457%20159.112C457.024%20158.55%20457.183%20158.122%20457.477%20157.829H457.917C458.944%20157.902%20460.166%20157.939%20461.584%20157.939C464.64%20157.939%20466.315%20157.902%20466.608%20157.829H466.975C467.122%20157.976%20467.195%20158.086%20467.195%20158.159C467.195%20158.232%20467.171%20158.452%20467.122%20158.819C466.999%20159.186%20466.889%20159.418%20466.792%20159.516H466.021C464.53%20159.589%20463.736%20159.797%20463.638%20160.139C464.371%20171.386%20464.787%20177.302%20464.885%20177.889L473.503%20162.706L473.429%20161.349C473.356%20160.323%20473.234%20159.772%20473.063%20159.699C472.916%20159.601%20472.403%20159.54%20471.522%20159.516C471.058%20159.516%20470.752%20159.491%20470.606%20159.442C470.459%20159.394%20470.386%20159.284%20470.386%20159.112C470.386%20159.063%20470.41%20158.917%20470.459%20158.672C470.483%20158.575%20470.508%20158.464%20470.532%20158.342C470.557%20158.22%20470.569%20158.134%20470.569%20158.086C470.569%20158.037%20470.606%20157.988%20470.679%20157.939C470.752%20157.89%20470.801%20157.866%20470.826%20157.866C470.85%20157.866%20470.948%20157.853%20471.119%20157.829Z%22%20fill%3D%22%23B8860B%22%2F%3E%3Cpath%20d%3D%22M492.238%20173.21C493.206%20171.81%20494.528%20171.11%20496.205%20171.11C497.034%20171.11%20497.864%20171.334%20498.694%20171.784C499.523%20172.233%20500.198%20173.028%20500.716%20174.169C501.356%20175.604%20501.675%20177.574%20501.675%20180.081C501.675%20182.76%20501.304%20184.808%20500.561%20186.225C500.215%20186.951%20499.74%20187.53%20499.135%20187.963C498.53%20188.395%20498.002%20188.663%20497.553%20188.766C497.104%20188.87%20496.663%20188.93%20496.231%20188.948C495.781%20188.948%20495.332%20188.896%20494.882%20188.792C494.433%20188.688%20493.906%20188.412%20493.301%20187.963C492.696%20187.513%20492.221%20186.934%20491.875%20186.225C491.132%20184.808%20490.76%20182.76%20490.76%20180.081C490.76%20177.073%20491.253%20174.783%20492.238%20173.21ZM498.072%20172.899C497.553%20172.345%20496.939%20172.069%20496.231%20172.069C495.505%20172.069%20494.882%20172.345%20494.364%20172.899C493.932%20173.348%20493.647%20173.97%20493.508%20174.765C493.37%20175.56%20493.301%20177.22%20493.301%20179.743C493.301%20182.474%20493.37%20184.272%20493.508%20185.136C493.647%20186.001%20493.958%20186.683%20494.442%20187.185C494.926%20187.703%20495.522%20187.963%20496.231%20187.963C496.922%20187.963%20497.51%20187.703%20497.994%20187.185C498.495%20186.666%20498.806%20185.94%20498.927%20185.007C499.048%20184.073%20499.117%20182.319%20499.135%20179.743C499.135%20177.237%20499.065%20175.586%20498.927%20174.791C498.789%20173.996%20498.504%20173.365%20498.072%20172.899Z%22%20fill%3D%22%23B8860B%22%2F%3E%3Cpath%20d%3D%22M517.814%20173.708C517.814%20170.726%20518.817%20168.281%20520.822%20166.374C522.826%20164.467%20525.198%20163.379%20527.936%20163.11C527.961%20163.11%20528.168%20163.11%20528.56%20163.11C528.951%20163.11%20529.464%20163.098%20530.1%20163.073C530.735%20163.049%20531.347%20163.049%20531.933%20163.073H535.564C535.931%20163.269%20536.114%20163.513%20536.114%20163.807C536.114%20164.027%20535.955%20164.259%20535.637%20164.504L531.823%20164.54C529.11%20164.565%20527.435%20164.638%20526.799%20164.76C525.601%20165.054%20524.538%20165.494%20523.609%20166.08C522.68%20166.667%20521.958%20167.266%20521.445%20167.877C520.932%20168.489%20520.492%20169.161%20520.125%20169.894C519.758%20170.628%20519.538%20171.215%20519.465%20171.655C519.391%20172.095%20519.33%20172.51%20519.281%20172.902V172.975H535.564C535.931%20173.17%20536.114%20173.415%20536.114%20173.708C536.114%20174.002%20535.931%20174.246%20535.564%20174.442H519.281V174.515C519.281%20175.029%20519.404%20175.652%20519.648%20176.385C519.893%20177.119%20520.296%20177.914%20520.858%20178.769C521.421%20179.625%20522.203%20180.407%20523.205%20181.116C524.208%20181.825%20525.393%20182.326%20526.763%20182.62C527.423%20182.767%20529.11%20182.852%20531.823%20182.877H535.637C535.955%20183.121%20536.114%20183.366%20536.114%20183.61C536.114%20183.903%20535.931%20184.148%20535.564%20184.343H532.007C531.836%20184.343%20531.53%20184.343%20531.09%20184.343C530.65%20184.343%20530.295%20184.356%20530.026%20184.38C526.237%20184.38%20523.242%20183.329%20521.042%20181.226C518.89%20179.221%20517.814%20176.716%20517.814%20173.708Z%22%20fill%3D%22%23B8860B%22%2F%3E%3Cpath%20d%3D%22M550.006%20158.489C550.006%20158.318%20550.14%20158.098%20550.409%20157.829H557.487C562.23%20157.878%20564.895%20157.951%20565.482%20158.049C566.019%20158.147%20566.533%20158.269%20567.022%20158.416C567.511%20158.562%20567.939%20158.721%20568.305%20158.892C568.672%20159.064%20569.014%20159.247%20569.332%20159.443C569.65%20159.638%20569.919%20159.834%20570.139%20160.029C570.359%20160.225%20570.579%20160.42%20570.799%20160.616C571.019%20160.812%20571.178%20160.995%20571.276%20161.166C571.374%20161.337%20571.496%20161.508%20571.643%20161.68C571.789%20161.851%20571.875%20162.01%20571.899%20162.156C571.924%20162.303%20571.973%20162.437%20572.046%20162.56L572.156%20162.78C572.278%20163.22%20572.339%20163.843%20572.339%20164.65C572.339%20165.872%20572.181%20166.826%20571.863%20167.511C571.007%20169.246%20569.296%20170.383%20566.728%20170.921L566.068%20171.104C566.24%20171.373%20566.46%20171.728%20566.728%20172.168C566.997%20172.608%20567.535%20173.415%20568.342%20174.588C569.149%20175.762%20569.895%20176.801%20570.579%20177.706C571.264%20178.61%20571.985%20179.454%20572.743%20180.236C573.501%20181.018%20574.1%20181.471%20574.54%20181.593C575.004%20181.715%20575.237%20181.947%20575.237%20182.29C575.237%20182.51%20575.09%20182.717%20574.796%20182.913H568.085C567.914%20182.742%20566.618%20180.786%20564.198%20177.045L560.604%20171.508H559.211V176.165C559.259%20179.001%20559.296%20180.517%20559.321%20180.713C559.394%20181.128%20559.638%20181.385%20560.054%20181.483C560.152%20181.507%20560.494%20181.544%20561.081%20181.593C561.692%20181.593%20562.108%20181.678%20562.328%20181.85C562.548%20182.119%20562.548%20182.424%20562.328%20182.766L562.071%20182.913H550.409C550.116%20182.766%20549.969%20182.559%20549.969%20182.29C549.969%20181.825%20550.446%20181.593%20551.399%20181.593C552.402%20181.52%20552.964%20181.312%20553.086%20180.969C553.208%20180.774%20553.27%20177.241%20553.27%20170.371C553.27%20163.501%20553.208%20159.944%20553.086%20159.699C552.94%20159.381%20552.377%20159.186%20551.399%20159.112C550.47%20159.112%20550.006%20158.905%20550.006%20158.489ZM558.22%20181.593C558.147%20181.422%20558.098%20181.299%20558.074%20181.226C558.049%20181.153%20558.025%20180.749%20558%20180.016C557.976%20179.282%20557.951%20178.243%20557.927%20176.899C557.903%20175.554%20557.903%20173.427%20557.927%20170.518V160.103L558.074%20159.589L558.33%20159.112H554.26C554.333%20159.284%20554.382%20159.406%20554.406%20159.479C554.431%20159.553%20554.455%20159.98%20554.48%20160.763C554.504%20161.545%20554.529%20162.596%20554.553%20163.917C554.578%20165.237%20554.578%20167.388%20554.553%20170.371C554.553%20175.505%20554.541%20178.586%20554.516%20179.613C554.492%20180.639%20554.419%20181.275%20554.296%20181.52C554.272%20181.544%20554.26%20181.568%20554.26%20181.593H558.22ZM566.142%20164.687C566.142%20163.611%20566.044%20162.706%20565.848%20161.973C565.653%20161.24%20565.359%20160.689%20564.968%20160.323C564.577%20159.956%20564.271%20159.724%20564.051%20159.626C563.831%20159.528%20563.55%20159.418%20563.208%20159.296C563.135%20159.296%20562.67%20159.235%20561.814%20159.112C560.861%20159.112%20560.152%20159.271%20559.687%20159.589C559.443%20159.76%20559.308%20160.066%20559.284%20160.506C559.259%20160.946%20559.235%20162.523%20559.211%20165.237V170.188H559.797C560.849%20170.188%20561.961%20170.114%20563.135%20169.968C564.259%20169.674%20565.042%20169.185%20565.482%20168.501C565.922%20167.816%20566.142%20166.545%20566.142%20164.687ZM567.425%20163.183C567.425%20163.183%20567.425%20163.684%20567.425%20164.687C567.425%20165.689%20567.376%20166.533%20567.279%20167.217C567.181%20167.902%20567.059%20168.366%20566.912%20168.611C566.765%20168.855%20566.655%20169.075%20566.582%20169.271C566.508%20169.466%20566.447%20169.589%20566.398%20169.638C566.398%20169.662%20566.411%20169.674%20566.435%20169.674C566.533%20169.674%20566.924%20169.552%20567.609%20169.308C569.956%20168.574%20571.129%20167.046%20571.129%20164.723C571.129%20162.352%20569.711%20160.702%20566.875%20159.773L566.142%20159.516L566.288%20159.919C566.606%20160.286%20566.936%20161.081%20567.279%20162.303C567.376%20162.89%20567.425%20163.684%20567.425%20164.687V163.183ZM566.398%20173.965C565.103%20172.156%20564.443%20171.251%20564.418%20171.251C564.271%20171.251%20564.161%20171.276%20564.088%20171.325C563.672%20171.325%20563.293%20171.361%20562.951%20171.435H562.071L568.672%20181.593H572.083L571.606%20181.043C570.31%20179.6%20568.574%20177.241%20566.398%20173.965Z%22%20fill%3D%22%23B8860B%22%2F%3E%3Cpath%20d%3D%22M576.456%20160.289C576.474%20160.237%20576.491%20160.168%20576.508%20160.082C576.525%20159.995%20576.569%20159.805%20576.638%20159.511C576.707%20159.218%20576.793%20158.95%20576.897%20158.708C577.001%20158.466%20577.13%20158.181%20577.286%20157.852C577.442%20157.524%20577.614%20157.264%20577.805%20157.074C577.995%20156.884%20578.219%20156.694%20578.479%20156.504C578.738%20156.314%20579.023%20156.236%20579.334%20156.271C580.631%20156.271%20581.46%20156.824%20581.823%20157.93L582.083%20157.697C583.154%20156.746%20584.174%20156.271%20585.142%20156.271C586.266%20156.271%20587.147%20156.677%20587.787%20157.489C588.426%20158.302%20588.746%20159.287%20588.746%20160.445C588.746%20162.312%20588.089%20164.032%20586.776%20165.604C585.462%20167.177%20583.975%20167.973%20582.316%20167.99C581.936%20167.99%20581.599%20167.938%20581.305%20167.834C581.115%20167.748%20580.916%20167.618%20580.708%20167.445C580.501%20167.272%20580.337%20167.126%20580.216%20167.005L580.034%20166.797C580.017%20166.814%20579.827%20167.566%20579.464%20169.053C579.101%20170.539%20578.919%20171.291%20578.919%20171.309C578.919%20171.412%20578.989%20171.473%20579.127%20171.49C579.265%20171.507%20579.585%20171.533%20580.086%20171.568H580.734C580.838%20171.689%20580.89%20171.767%20580.89%20171.801C580.89%20171.836%20580.864%20171.991%20580.812%20172.268C580.76%20172.475%20580.708%20172.605%20580.657%20172.657C580.605%20172.709%20580.492%20172.743%20580.32%20172.76C580.285%20172.76%20580.19%20172.76%20580.034%20172.76C579.879%20172.76%20579.585%20172.752%20579.153%20172.735C578.721%20172.717%20578.185%20172.709%20577.545%20172.709C576.335%20172.709%20575.566%20172.726%20575.238%20172.76H575.03C574.909%20172.64%20574.849%20172.544%20574.849%20172.475C574.883%20172.009%20574.996%20171.706%20575.186%20171.568H575.704C576.292%20171.551%20576.629%20171.447%20576.716%20171.257C576.767%20171.153%20577.295%20169.087%20578.297%20165.06C579.3%20161.033%20579.827%20158.898%20579.879%20158.656C579.896%20158.569%20579.905%20158.448%20579.905%20158.293C579.905%20157.584%20579.689%20157.23%20579.257%20157.23C578.876%20157.23%20578.556%20157.472%20578.297%20157.956C578.038%20158.44%20577.839%20158.967%20577.701%20159.537C577.563%20160.108%20577.459%20160.419%20577.39%20160.471C577.355%20160.505%20577.217%20160.523%20576.975%20160.523H576.612C576.508%20160.419%20576.456%20160.341%20576.456%20160.289ZM580.475%20165.086C580.855%20166.4%20581.495%20167.056%20582.394%20167.056C582.912%20167.056%20583.414%20166.858%20583.898%20166.46C584.382%20166.063%20584.779%20165.561%20585.09%20164.956C585.401%20164.369%20585.712%20163.435%20586.024%20162.156C586.335%20160.877%20586.499%20159.926%20586.516%20159.304V159.148C586.516%20157.869%20585.989%20157.23%20584.935%20157.23C584.745%20157.23%20584.554%20157.256%20584.364%20157.308C584.174%20157.359%20583.984%20157.437%20583.794%20157.541C583.604%20157.645%20583.431%20157.757%20583.275%20157.878C583.12%20157.999%20582.973%20158.129%20582.835%20158.267C582.696%20158.405%20582.567%20158.535%20582.446%20158.656C582.325%20158.777%20582.23%20158.898%20582.16%20159.019C582.091%20159.14%20582.022%20159.226%20581.953%20159.278L581.875%20159.382C581.875%20159.399%20581.849%20159.52%20581.797%20159.745C581.746%20159.97%20581.65%20160.35%20581.512%20160.886C581.374%20161.421%20581.253%20161.871%20581.149%20162.234C580.7%20164.066%20580.475%20165.017%20580.475%20165.086Z%22%20fill%3D%22%23B8860B%22%2F%3E%3Cpath%20d%3D%22M605.236%20166.979C605.236%20167.324%20605.055%20167.497%20604.692%20167.497C604.605%20167.497%20604.415%20167.359%20604.121%20167.082C603.827%20166.806%20603.015%20166.002%20601.684%20164.671L598.987%20161.975L596.265%20164.697C594.433%20166.495%20593.491%20167.411%20593.439%20167.445C593.37%20167.48%20593.309%20167.497%20593.257%20167.497C593.119%20167.497%20592.998%20167.445%20592.894%20167.342C592.791%20167.238%20592.73%20167.117%20592.713%20166.979C592.713%20166.858%20592.834%20166.668%20593.076%20166.408C593.318%20166.149%20594.13%20165.328%20595.513%20163.945L598.21%20161.249L595.513%20158.552C594.113%20157.152%20593.301%20156.331%20593.076%20156.089C592.851%20155.847%20592.73%20155.657%20592.713%20155.519C592.713%20155.363%20592.765%20155.242%20592.869%20155.156C592.972%20155.069%20593.102%20155.026%20593.257%20155.026C593.344%20155.026%20593.404%20155.035%20593.439%20155.052C593.491%20155.087%20594.433%20156.003%20596.265%20157.8L598.987%20160.523L601.684%20157.826C603.015%20156.495%20603.827%20155.692%20604.121%20155.415C604.415%20155.138%20604.605%20155%20604.692%20155C605.055%20155%20605.236%20155.173%20605.236%20155.519C605.236%20155.64%20605.15%20155.795%20604.977%20155.985C604.804%20156.176%20603.957%20157.023%20602.436%20158.526L599.739%20161.249L602.436%20163.971C603.888%20165.423%20604.717%20166.261%20604.925%20166.486C605.132%20166.711%20605.236%20166.875%20605.236%20166.979Z%22%20fill%3D%22%23B8860B%22%2F%3E%3Cpath%20d%3D%22M609.929%20163.66C609.929%20161.914%20610.586%20160.255%20611.899%20158.682C613.213%20157.109%20614.691%20156.314%20616.333%20156.296C617.37%20156.296%20618.165%20156.746%20618.718%20157.645C618.736%20157.627%20618.779%20157.576%20618.848%20157.489C618.917%20157.403%20619.012%20157.308%20619.133%20157.204C619.254%20157.1%20619.392%20156.997%20619.548%20156.893C620.136%20156.478%20620.533%20156.271%20620.741%20156.271C620.81%20156.271%20620.87%20156.296%20620.922%20156.348C620.974%20156.4%20621%20156.443%20621%20156.478C621%20156.634%20620.404%20159.097%20619.211%20163.867C617.984%20168.776%20617.37%20171.265%20617.37%20171.334C617.37%20171.473%20617.759%20171.551%20618.537%20171.568H619.185C619.289%20171.706%20619.341%20171.784%20619.341%20171.801C619.341%20171.818%20619.315%20171.974%20619.263%20172.268C619.194%20172.493%20619.107%20172.657%20619.004%20172.76H618.667C618.131%20172.726%20617.197%20172.709%20615.866%20172.709C615.382%20172.709%20614.942%20172.709%20614.544%20172.709C614.146%20172.709%20613.827%20172.709%20613.585%20172.709C613.343%20172.709%20613.196%20172.717%20613.144%20172.735C612.902%20172.735%20612.781%20172.665%20612.781%20172.527C612.781%20172.475%20612.798%20172.346%20612.833%20172.138C612.902%20171.862%20612.962%20171.697%20613.014%20171.646C613.066%20171.594%20613.239%20171.568%20613.533%20171.568C614.363%20171.568%20614.864%20171.507%20615.037%20171.386C615.175%20171.3%20615.417%20170.539%20615.763%20169.105C616.108%20167.67%20616.29%20166.935%20616.307%20166.901C616.307%20166.884%20616.247%20166.909%20616.126%20166.979C615.296%20167.653%20614.432%20167.99%20613.533%20167.99C612.582%20167.99%20611.744%20167.644%20611.018%20166.953C610.292%20166.261%20609.929%20165.164%20609.929%20163.66ZM618.2%20159.278C617.802%20157.913%20617.154%20157.23%20616.255%20157.23C615.65%20157.23%20615.071%20157.498%20614.518%20158.034C613.965%20158.569%20613.533%20159.269%20613.222%20160.134C612.72%20161.499%20612.366%20162.917%20612.159%20164.386C612.159%20164.438%20612.159%20164.533%20612.159%20164.671C612.159%20164.809%20612.15%20164.913%20612.133%20164.982C612.133%20165.76%20612.288%20166.296%20612.599%20166.59C612.911%20166.884%20613.282%20167.039%20613.714%20167.056C614.648%20167.056%20615.607%20166.434%20616.592%20165.19L616.8%20164.904L618.2%20159.278Z%22%20fill%3D%22%23B8860B%22%2F%3E%3Cpath%20d%3D%22M380.426%20405.556C380.426%20405.965%20380.819%20406.204%20381.603%20406.272C381.91%20406.272%20382.064%20406.366%20382.064%20406.553C382.064%20406.571%20382.047%20406.681%20382.013%20406.886C381.962%20407.125%20381.91%20407.278%20381.859%20407.347C381.808%20407.415%20381.697%20407.449%20381.527%20407.449C381.492%20407.449%20381.39%20407.449%20381.22%20407.449C381.049%20407.449%20380.759%20407.44%20380.35%20407.423C379.94%20407.406%20379.394%20407.398%20378.712%20407.398C377.501%20407.398%20376.725%20407.415%20376.384%20407.449H376.179C376.06%20407.33%20376%20407.236%20376%20407.168C376.034%20406.707%20376.145%20406.408%20376.333%20406.272H376.691C377.885%20406.221%20378.755%20405.794%20379.301%20404.993C379.403%20404.89%20380.947%20402.289%20383.932%20397.189C386.917%20392.089%20388.435%20389.505%20388.486%20389.436C388.605%20389.232%20388.81%20389.129%20389.1%20389.129H389.305H389.74C389.842%20389.283%20389.893%20389.368%20389.893%20389.385L390.61%20397.649C391.087%20403.159%20391.343%20405.939%20391.377%20405.991C391.463%20406.178%20392.017%20406.272%20393.04%20406.272C393.467%20406.272%20393.68%20406.357%20393.68%20406.528C393.68%20406.562%20393.654%20406.681%20393.603%20406.886C393.552%20407.142%20393.509%20407.295%20393.475%20407.347C393.441%20407.398%20393.322%20407.432%20393.117%20407.449C393.066%20407.449%20392.947%20407.449%20392.759%20407.449C392.571%20407.449%20392.239%20407.44%20391.761%20407.423C391.283%20407.406%20390.661%20407.398%20389.893%20407.398C388.29%20407.398%20387.241%20407.415%20386.746%20407.449H386.439C386.337%20407.347%20386.286%20407.27%20386.286%20407.219C386.286%20407.168%20386.303%20407.014%20386.337%20406.758C386.405%20406.536%20386.456%20406.4%20386.49%20406.349L386.593%20406.272H387.104C388.026%20406.221%20388.571%20406.076%20388.742%20405.837L388.435%20401.999H382.345L381.475%20403.483C380.776%20404.66%20380.426%20405.351%20380.426%20405.556ZM388.307%20400.797C388.307%20400.609%20388.205%20399.278%20388%20396.805C387.795%20394.332%20387.676%20393.087%20387.642%20393.07L386.951%20394.17C386.678%20394.63%20386.141%20395.543%20385.339%20396.907L383.036%20400.797L385.672%20400.822C387.428%20400.822%20388.307%20400.814%20388.307%20400.797Z%22%20fill%3D%22%23B8860B%22%2F%3E%3Cpath%20d%3D%22M403.551%20401.052C403.551%20398.971%20404.251%20397.266%20405.649%20395.935C407.048%20394.605%20408.703%20393.846%20410.613%20393.658C410.63%20393.658%20410.775%20393.658%20411.048%20393.658C411.321%20393.658%20411.679%20393.65%20412.123%20393.632C412.566%20393.615%20412.993%20393.615%20413.402%20393.632H415.935C416.191%20393.769%20416.319%20393.94%20416.319%20394.144C416.319%20394.298%20416.208%20394.46%20415.986%20394.63L413.325%20394.656C411.432%20394.673%20410.263%20394.724%20409.82%20394.809C408.984%20395.014%20408.242%20395.321%20407.594%20395.731C406.946%20396.14%20406.442%20396.558%20406.084%20396.984C405.726%20397.411%20405.419%20397.88%20405.163%20398.392C404.907%20398.903%20404.754%20399.313%20404.703%20399.62C404.651%20399.927%20404.609%20400.217%20404.575%20400.49V400.541H415.935C416.191%20400.677%20416.319%20400.848%20416.319%20401.052C416.319%20401.257%20416.191%20401.428%20415.935%20401.564H404.575V401.615C404.575%20401.974%20404.66%20402.409%20404.831%20402.92C405.001%20403.432%20405.283%20403.986%20405.675%20404.583C406.067%20405.18%20406.613%20405.726%20407.312%20406.221C408.012%20406.716%20408.839%20407.065%20409.794%20407.27C410.255%20407.372%20411.432%20407.432%20413.325%20407.449H415.986C416.208%20407.62%20416.319%20407.79%20416.319%20407.961C416.319%20408.165%20416.191%20408.336%20415.935%20408.472H413.453C413.334%20408.472%20413.12%20408.472%20412.813%20408.472C412.506%20408.472%20412.259%20408.481%20412.071%20408.498C409.427%20408.498%20407.338%20407.765%20405.803%20406.298C404.302%20404.899%20403.551%20403.151%20403.551%20401.052Z%22%20fill%3D%22%23B8860B%22%2F%3E%3Cpath%20d%3D%22M426.011%20390.434C426.011%20390.315%20426.105%20390.161%20426.292%20389.974H431.23C434.539%20390.008%20436.399%20390.059%20436.808%20390.127C437.183%20390.195%20437.542%20390.281%20437.883%20390.383C438.224%20390.485%20438.522%20390.596%20438.778%20390.716C439.034%20390.835%20439.273%20390.963%20439.495%20391.099C439.716%20391.236%20439.904%20391.372%20440.058%20391.509C440.211%20391.645%20440.365%20391.782%20440.518%20391.918C440.672%20392.055%20440.782%20392.183%20440.851%20392.302C440.919%20392.421%20441.004%20392.541%20441.107%20392.66C441.209%20392.78%20441.269%20392.89%20441.286%20392.993C441.303%20393.095%20441.337%20393.189%20441.388%20393.274L441.465%20393.428C441.55%20393.735%20441.593%20394.17%20441.593%20394.733C441.593%20395.586%20441.482%20396.251%20441.26%20396.728C440.663%20397.939%20439.469%20398.733%20437.678%20399.108L437.217%20399.236C437.337%20399.423%20437.49%20399.671%20437.678%20399.978C437.866%20400.285%20438.241%20400.848%20438.804%20401.667C439.367%20402.485%20439.887%20403.21%20440.365%20403.841C440.842%20404.472%20441.345%20405.061%20441.874%20405.607C442.403%20406.153%20442.821%20406.468%20443.128%20406.554C443.452%20406.639%20443.614%20406.801%20443.614%20407.04C443.614%20407.193%20443.512%20407.338%20443.307%20407.475H438.625C438.505%20407.355%20437.601%20405.991%20435.913%20403.381L433.405%20399.517H432.433V402.767C432.467%20404.745%20432.493%20405.803%20432.51%20405.939C432.561%20406.229%20432.731%20406.409%20433.021%20406.477C433.09%20406.494%20433.328%20406.519%20433.738%20406.554C434.164%20406.554%20434.454%20406.613%20434.608%20406.733C434.761%20406.92%20434.761%20407.133%20434.608%20407.372L434.429%20407.475H426.292C426.087%20407.372%20425.985%20407.227%20425.985%20407.04C425.985%20406.716%20426.318%20406.554%20426.983%20406.554C427.682%20406.502%20428.075%20406.357%20428.16%20406.119C428.245%20405.982%20428.288%20403.517%20428.288%20398.724C428.288%20393.931%20428.245%20391.449%20428.16%20391.279C428.058%20391.057%20427.665%20390.92%20426.983%20390.869C426.335%20390.869%20426.011%20390.724%20426.011%20390.434ZM431.742%20406.554C431.691%20406.434%20431.657%20406.349%20431.64%20406.298C431.623%20406.246%20431.606%20405.965%20431.589%20405.453C431.571%20404.942%20431.554%20404.217%20431.537%20403.278C431.52%20402.34%20431.52%20400.856%20431.537%20398.826V391.56L431.64%20391.202L431.819%20390.869H428.979C429.03%20390.989%20429.064%20391.074%20429.081%20391.125C429.098%20391.176%20429.115%20391.475%20429.132%20392.021C429.149%20392.566%20429.166%20393.3%20429.183%20394.221C429.2%20395.142%20429.2%20396.643%20429.183%20398.724C429.183%20402.306%20429.175%20404.455%20429.158%20405.172C429.141%20405.888%20429.09%20406.332%20429.004%20406.502C428.987%20406.519%20428.979%20406.536%20428.979%20406.554H431.742ZM437.269%20394.758C437.269%20394.008%20437.2%20393.377%20437.064%20392.865C436.928%20392.353%20436.723%20391.969%20436.45%20391.714C436.177%20391.458%20435.964%20391.296%20435.81%20391.227C435.657%20391.159%20435.461%20391.082%20435.222%20390.997C435.171%20390.997%20434.846%20390.954%20434.249%20390.869C433.584%20390.869%20433.09%20390.98%20432.765%20391.202C432.595%20391.321%20432.501%20391.534%20432.484%20391.841C432.467%20392.148%20432.45%20393.249%20432.433%20395.142V398.596H432.842C433.576%20398.596%20434.352%20398.545%20435.171%20398.443C435.955%20398.238%20436.501%20397.897%20436.808%20397.419C437.115%20396.942%20437.269%20396.055%20437.269%20394.758ZM438.164%20393.709C438.164%20393.709%20438.164%20394.059%20438.164%20394.758C438.164%20395.458%20438.13%20396.046%20438.062%20396.524C437.994%20397.001%20437.908%20397.325%20437.806%20397.496C437.704%20397.667%20437.627%20397.82%20437.576%20397.957C437.525%20398.093%20437.482%20398.178%20437.448%20398.212C437.448%20398.229%20437.456%20398.238%20437.473%20398.238C437.542%20398.238%20437.814%20398.153%20438.292%20397.982C439.93%20397.47%20440.748%20396.404%20440.748%20394.784C440.748%20393.129%20439.759%20391.978%20437.78%20391.33L437.269%20391.151L437.371%20391.432C437.593%20391.688%20437.823%20392.242%20438.062%20393.095C438.13%20393.505%20438.164%20394.059%20438.164%20394.758V393.709ZM437.448%20401.232C436.544%20399.969%20436.083%20399.338%20436.066%20399.338C435.964%20399.338%20435.887%20399.355%20435.836%20399.389C435.546%20399.389%20435.281%20399.415%20435.043%20399.466H434.429L439.034%20406.554H441.414L441.081%20406.17C440.177%20405.163%20438.966%20403.517%20437.448%20401.232Z%22%20fill%3D%22%23B8860B%22%2F%3E%3Cpath%20d%3D%22M444.465%20391.69C444.477%20391.654%20444.489%20391.606%20444.501%20391.545C444.513%20391.485%20444.543%20391.352%20444.592%20391.147C444.64%20390.942%20444.7%20390.755%20444.773%20390.587C444.845%20390.418%20444.935%20390.219%20445.044%20389.99C445.152%20389.761%20445.273%20389.58%20445.406%20389.447C445.538%20389.314%20445.695%20389.182%20445.876%20389.049C446.057%20388.916%20446.256%20388.862%20446.473%20388.886C447.377%20388.886%20447.956%20389.272%20448.21%20390.044L448.39%20389.881C449.138%20389.218%20449.85%20388.886%20450.525%20388.886C451.309%20388.886%20451.924%20389.17%20452.37%20389.736C452.816%20390.303%20453.039%20390.991%20453.039%20391.799C453.039%20393.101%20452.581%20394.301%20451.665%20395.398C450.748%20396.496%20449.711%20397.051%20448.553%20397.063C448.288%20397.063%20448.053%20397.026%20447.848%20396.954C447.715%20396.894%20447.576%20396.803%20447.432%20396.683C447.287%20396.562%20447.172%20396.46%20447.088%20396.375L446.961%20396.231C446.949%20396.243%20446.817%20396.767%20446.563%20397.804C446.31%20398.841%20446.184%20399.366%20446.184%20399.378C446.184%20399.45%20446.232%20399.493%20446.328%20399.505C446.425%20399.517%20446.648%20399.535%20446.998%20399.559H447.45C447.522%20399.643%20447.558%20399.698%20447.558%20399.722C447.558%20399.746%20447.54%20399.854%20447.504%20400.047C447.468%20400.192%20447.432%20400.283%20447.396%20400.319C447.359%20400.355%20447.281%20400.379%20447.16%20400.391C447.136%20400.391%20447.07%20400.391%20446.961%20400.391C446.853%20400.391%20446.648%20400.385%20446.346%20400.373C446.045%20400.361%20445.671%20400.355%20445.225%20400.355C444.381%20400.355%20443.844%20400.367%20443.615%20400.391H443.47C443.386%20400.307%20443.344%20400.24%20443.344%20400.192C443.368%20399.867%20443.446%20399.655%20443.579%20399.559H443.94C444.35%20399.547%20444.586%20399.475%20444.646%20399.342C444.682%20399.27%20445.05%20397.828%20445.749%20395.019C446.449%20392.209%20446.817%20390.719%20446.853%20390.55C446.865%20390.49%20446.871%20390.406%20446.871%20390.297C446.871%20389.803%20446.72%20389.556%20446.419%20389.556C446.153%20389.556%20445.93%20389.724%20445.749%20390.062C445.569%20390.4%20445.43%20390.768%20445.333%20391.166C445.237%20391.563%20445.165%20391.781%20445.116%20391.817C445.092%20391.841%20444.996%20391.853%20444.827%20391.853H444.574C444.501%20391.781%20444.465%20391.726%20444.465%20391.69ZM447.269%20395.037C447.534%20395.953%20447.98%20396.411%20448.608%20396.411C448.969%20396.411%20449.319%20396.273%20449.657%20395.995C449.994%20395.718%20450.272%20395.368%20450.489%20394.946C450.706%20394.536%20450.923%20393.885%20451.14%20392.993C451.357%20392.1%20451.472%20391.437%20451.484%20391.003V390.894C451.484%20390.002%20451.116%20389.556%20450.38%20389.556C450.248%20389.556%20450.115%20389.574%20449.982%20389.61C449.85%20389.646%20449.717%20389.7%20449.584%20389.773C449.452%20389.845%20449.331%20389.923%20449.223%20390.008C449.114%20390.092%20449.012%20390.183%20448.915%20390.279C448.819%20390.376%20448.728%20390.466%20448.644%20390.55C448.559%20390.635%20448.493%20390.719%20448.445%20390.804C448.396%20390.888%20448.348%20390.948%20448.3%20390.985L448.246%20391.057C448.246%20391.069%20448.228%20391.153%20448.191%20391.31C448.155%20391.467%20448.089%20391.732%20447.992%20392.106C447.896%20392.48%20447.812%20392.794%20447.739%20393.047C447.426%20394.325%20447.269%20394.988%20447.269%20395.037Z%22%20fill%3D%22%23B8860B%22%2F%3E%3Cpath%20d%3D%22M464.544%20396.357C464.544%20396.598%20464.418%20396.719%20464.164%20396.719C464.104%20396.719%20463.971%20396.623%20463.766%20396.43C463.561%20396.237%20462.995%20395.676%20462.066%20394.747L460.185%20392.866L458.285%20394.765C457.007%20396.02%20456.35%20396.659%20456.314%20396.683C456.265%20396.707%20456.223%20396.719%20456.187%20396.719C456.091%20396.719%20456.006%20396.683%20455.934%20396.61C455.861%20396.538%20455.819%20396.454%20455.807%20396.357C455.807%20396.273%20455.892%20396.14%20456.06%20395.959C456.229%20395.778%20456.796%20395.206%20457.761%20394.241L459.642%20392.359L457.761%20390.478C456.784%20389.501%20456.217%20388.928%20456.06%20388.76C455.904%20388.591%20455.819%20388.458%20455.807%20388.362C455.807%20388.253%20455.843%20388.169%20455.916%20388.108C455.988%20388.048%20456.078%20388.018%20456.187%20388.018C456.247%20388.018%20456.29%20388.024%20456.314%20388.036C456.35%20388.06%20457.007%20388.699%20458.285%20389.954L460.185%20391.853L462.066%20389.972C462.995%20389.043%20463.561%20388.482%20463.766%20388.289C463.971%20388.096%20464.104%20388%20464.164%20388C464.418%20388%20464.544%20388.12%20464.544%20388.362C464.544%20388.446%20464.484%20388.555%20464.363%20388.687C464.243%20388.82%20463.652%20389.411%20462.591%20390.46L460.709%20392.359L462.591%20394.259C463.604%20395.272%20464.183%20395.857%20464.327%20396.014C464.472%20396.17%20464.544%20396.285%20464.544%20396.357Z%22%20fill%3D%22%23B8860B%22%2F%3E%3Cpath%20d%3D%22M467.601%20391.69C467.613%20391.654%20467.626%20391.606%20467.638%20391.545C467.65%20391.485%20467.68%20391.352%20467.728%20391.147C467.776%20390.942%20467.837%20390.756%20467.909%20390.587C467.981%20390.418%20468.072%20390.219%20468.18%20389.99C468.289%20389.761%20468.409%20389.58%20468.542%20389.447C468.675%20389.314%20468.831%20389.182%20469.012%20389.049C469.193%20388.916%20469.392%20388.862%20469.609%20388.886C469.959%20388.886%20470.267%20388.959%20470.532%20389.103C470.797%20389.248%20470.984%20389.381%20471.093%20389.501C471.201%20389.622%20471.268%20389.743%20471.292%20389.863C471.316%20389.935%20471.334%20389.972%20471.346%20389.972C471.358%20389.972%20471.406%20389.929%20471.491%20389.845C472.07%20389.218%20472.739%20388.904%20473.499%20388.904H473.553C474.132%20388.904%20474.566%20389.152%20474.855%20389.646C474.952%20389.875%20475%20390.098%20475%20390.315C475%20390.677%20474.922%20390.967%20474.765%20391.184C474.608%20391.401%20474.451%20391.539%20474.295%20391.6C474.138%20391.66%20473.987%20391.684%20473.842%20391.672C473.601%20391.672%20473.408%20391.606%20473.263%20391.473C473.119%20391.34%20473.046%20391.166%20473.046%20390.948C473.046%20390.37%20473.384%20389.984%20474.059%20389.791C473.794%20389.634%20473.577%20389.556%20473.408%20389.556C472.733%20389.556%20472.112%20390.002%20471.545%20390.894C471.424%20391.087%20471.334%20391.286%20471.274%20391.491C471.213%20391.696%20471.008%20392.498%20470.659%20393.897C470.249%20395.585%20470.013%20396.484%20469.953%20396.592C469.796%20396.918%20469.537%20397.081%20469.175%20397.081C469.018%20397.081%20468.892%20397.045%20468.795%20396.972C468.699%20396.9%20468.633%20396.828%20468.596%20396.755C468.56%20396.683%20468.548%20396.623%20468.56%20396.574C468.56%20396.418%20468.801%20395.38%20469.284%20393.463C469.766%20391.545%20470.007%20390.478%20470.007%20390.261C470.007%20389.791%20469.851%20389.556%20469.537%20389.556C469.284%20389.556%20469.067%20389.724%20468.886%20390.062C468.705%20390.4%20468.56%20390.768%20468.452%20391.166C468.343%20391.564%20468.277%20391.781%20468.253%20391.817C468.229%20391.841%20468.132%20391.853%20467.963%20391.853H467.71C467.638%20391.781%20467.601%20391.726%20467.601%20391.69Z%22%20fill%3D%22%23B8860B%22%2F%3E%3Cpath%20d%3D%22M609.82%20391.028C609.378%20391.028%20609.115%20391.02%20609.033%20391.004C608.951%20390.987%20608.91%20390.897%20608.91%20390.733C608.91%20390.291%20609.001%20390.02%20609.181%20389.922C609.197%20389.905%20610.263%20389.897%20612.378%20389.897C616.625%20389.897%20618.863%20389.913%20619.092%20389.946C620.125%20390.11%20620.986%20390.479%20621.675%20391.053C622.363%20391.627%20622.716%20392.381%20622.732%20393.316C622.732%20394.365%20622.257%20395.308%20621.306%20396.144C620.355%20396.98%20619.265%20397.546%20618.035%20397.841L617.789%20397.915C618.838%20398.062%20619.699%20398.439%20620.371%20399.046C621.044%20399.653%20621.388%20400.407%20621.404%20401.309C621.404%20402.506%20620.814%20403.662%20619.633%20404.776C618.453%20405.891%20617.002%20406.523%20615.28%20406.67C615.166%20406.687%20613.436%20406.695%20610.091%20406.695C606.812%20406.695%20605.148%20406.678%20605.098%20406.646C605.033%20406.596%20605%20406.531%20605%20406.449C605%20406.334%20605.016%20406.219%20605.049%20406.105C605.131%20405.793%20605.213%20405.621%20605.295%20405.588C605.361%20405.572%20605.484%20405.563%20605.664%20405.563H605.812C606.254%20405.563%20606.746%20405.539%20607.287%20405.49C607.517%20405.441%20607.673%20405.342%20607.755%20405.195C607.804%20405.129%20608.394%20402.85%20609.525%20398.357C610.657%20393.865%20611.222%20391.496%20611.222%20391.25C611.222%20391.135%20610.755%20391.061%20609.82%20391.028ZM620.101%20393.316C620.101%20392.824%20619.978%20392.365%20619.732%20391.938C619.486%20391.512%20619.084%20391.233%20618.527%20391.102C618.412%20391.069%20617.658%20391.045%20616.264%20391.028C615.936%20391.028%20615.592%20391.028%20615.231%20391.028C614.87%20391.028%20614.583%20391.037%20614.37%20391.053H614.051C613.805%20391.069%20613.649%20391.151%20613.583%20391.299C613.551%20391.365%20613.288%20392.348%20612.796%20394.25C612.796%20394.299%20612.788%20394.349%20612.772%20394.398L612.009%20397.497H614.001C615.33%20397.497%20616.092%20397.48%20616.289%20397.447C617.272%20397.284%20618.15%20396.808%20618.92%20396.021C619.691%20395.234%20620.084%20394.332%20620.101%20393.316ZM618.773%20401.063C618.773%20400.341%20618.584%20399.743%20618.207%20399.267C617.83%20398.792%20617.338%20398.513%20616.731%20398.431C616.633%20398.415%20615.953%20398.407%20614.69%20398.407C612.755%20398.407%20611.78%20398.415%20611.763%20398.431C611.763%20398.464%20611.624%20399.062%20611.345%20400.227C611.067%20401.391%20610.78%20402.547%20610.484%20403.694L610.042%20405.416C610.042%20405.482%20610.14%20405.514%20610.337%20405.514C610.534%20405.514%20611.198%20405.531%20612.329%20405.563C613.788%20405.563%20614.575%20405.555%20614.69%20405.539C615.707%20405.424%20616.641%20404.949%20617.494%20404.112C618.346%20403.276%20618.773%20402.26%20618.773%20401.063Z%22%20fill%3D%22%23B8860B%22%2F%3E%3Cpath%20d%3D%22M631.704%20400.546C631.704%20398.546%20632.376%20396.906%20633.721%20395.627C635.065%20394.349%20636.656%20393.619%20638.492%20393.439C638.509%20393.439%20638.648%20393.439%20638.91%20393.439C639.173%20393.439%20639.517%20393.43%20639.943%20393.414C640.37%20393.398%20640.779%20393.398%20641.173%20393.414H643.608C643.854%20393.545%20643.977%20393.709%20643.977%20393.906C643.977%20394.053%20643.87%20394.209%20643.657%20394.373L641.099%20394.398C639.279%20394.414%20638.156%20394.463%20637.73%20394.545C636.926%20394.742%20636.213%20395.037%20635.59%20395.431C634.967%20395.824%20634.483%20396.226%20634.139%20396.636C633.795%20397.046%20633.5%20397.497%20633.254%20397.988C633.008%20398.48%20632.86%20398.874%20632.811%20399.169C632.762%20399.464%20632.721%20399.743%20632.688%20400.005V400.054H643.608C643.854%20400.186%20643.977%20400.35%20643.977%20400.546C643.977%20400.743%20643.854%20400.907%20643.608%20401.038H632.688V401.087C632.688%20401.432%20632.77%20401.85%20632.934%20402.342C633.098%20402.834%20633.368%20403.366%20633.746%20403.94C634.123%20404.514%20634.647%20405.039%20635.32%20405.514C635.992%20405.99%20636.787%20406.326%20637.705%20406.523C638.148%20406.621%20639.279%20406.678%20641.099%20406.695H643.657C643.87%20406.859%20643.977%20407.023%20643.977%20407.187C643.977%20407.383%20643.854%20407.547%20643.608%20407.679H641.222C641.107%20407.679%20640.902%20407.679%20640.607%20407.679C640.312%20407.679%20640.074%20407.687%20639.894%20407.703C637.353%20407.703%20635.344%20406.998%20633.869%20405.588C632.426%20404.244%20631.704%20402.563%20631.704%20400.546Z%22%20fill%3D%22%23B8860B%22%2F%3E%3Cpath%20d%3D%22M653.293%20390.34C653.293%20390.225%20653.383%20390.077%20653.563%20389.897H658.31C661.491%20389.93%20663.278%20389.979%20663.672%20390.044C664.032%20390.11%20664.377%20390.192%20664.705%20390.29C665.032%20390.389%20665.319%20390.495%20665.565%20390.61C665.811%20390.725%20666.041%20390.848%20666.254%20390.979C666.467%20391.11%20666.647%20391.241%20666.795%20391.373C666.943%20391.504%20667.09%20391.635%20667.238%20391.766C667.385%20391.897%20667.492%20392.02%20667.557%20392.135C667.623%20392.25%20667.705%20392.365%20667.803%20392.479C667.902%20392.594%20667.959%20392.701%20667.976%20392.799C667.992%20392.897%20668.025%20392.988%20668.074%20393.07L668.148%20393.217C668.23%20393.512%20668.271%20393.93%20668.271%20394.471C668.271%20395.291%20668.164%20395.931%20667.951%20396.39C667.377%20397.554%20666.229%20398.316%20664.508%20398.677L664.065%20398.8C664.18%20398.98%20664.327%20399.218%20664.508%20399.513C664.688%20399.808%20665.049%20400.349%20665.59%20401.136C666.131%20401.923%20666.631%20402.62%20667.09%20403.227C667.549%20403.834%20668.033%20404.399%20668.541%20404.924C669.049%20405.449%20669.451%20405.752%20669.746%20405.834C670.058%20405.916%20670.214%20406.072%20670.214%20406.301C670.214%20406.449%20670.115%20406.588%20669.918%20406.719H665.418C665.303%20406.604%20664.434%20405.293%20662.811%20402.784L660.401%20399.07H659.466V402.194C659.499%20404.096%20659.523%20405.112%20659.54%20405.244C659.589%20405.522%20659.753%20405.694%20660.032%20405.76C660.097%20405.776%20660.327%20405.801%20660.72%20405.834C661.13%20405.834%20661.409%20405.891%20661.556%20406.006C661.704%20406.186%20661.704%20406.391%20661.556%20406.621L661.384%20406.719H653.563C653.367%20406.621%20653.268%20406.481%20653.268%20406.301C653.268%20405.99%20653.588%20405.834%20654.227%20405.834C654.9%20405.785%20655.277%20405.645%20655.359%20405.416C655.441%20405.285%20655.482%20402.915%20655.482%20398.308C655.482%20393.701%20655.441%20391.315%20655.359%20391.151C655.26%20390.938%20654.883%20390.807%20654.227%20390.758C653.604%20390.758%20653.293%20390.618%20653.293%20390.34ZM658.802%20405.834C658.753%20405.719%20658.72%20405.637%20658.704%20405.588C658.687%20405.539%20658.671%20405.268%20658.654%20404.776C658.638%20404.284%20658.622%20403.588%20658.605%20402.686C658.589%20401.784%20658.589%20400.358%20658.605%20398.406V391.422L658.704%20391.077L658.876%20390.758H656.146C656.195%20390.872%20656.228%20390.954%20656.244%20391.004C656.261%20391.053%20656.277%20391.34%20656.293%20391.864C656.31%20392.389%20656.326%20393.094%20656.343%20393.98C656.359%20394.865%20656.359%20396.308%20656.343%20398.308C656.343%20401.751%20656.334%20403.817%20656.318%20404.506C656.302%20405.194%20656.252%20405.621%20656.17%20405.785C656.154%20405.801%20656.146%20405.817%20656.146%20405.834H658.802ZM664.114%20394.496C664.114%20393.775%20664.049%20393.168%20663.918%20392.676C663.786%20392.184%20663.59%20391.815%20663.327%20391.569C663.065%20391.323%20662.86%20391.168%20662.712%20391.102C662.565%20391.036%20662.376%20390.963%20662.147%20390.881C662.098%20390.881%20661.786%20390.84%20661.212%20390.758C660.573%20390.758%20660.097%20390.864%20659.786%20391.077C659.622%20391.192%20659.532%20391.397%20659.515%20391.692C659.499%20391.987%20659.482%20393.045%20659.466%20394.865V398.185H659.859C660.565%20398.185%20661.311%20398.136%20662.098%20398.038C662.852%20397.841%20663.376%20397.513%20663.672%20397.054C663.967%20396.595%20664.114%20395.742%20664.114%20394.496ZM664.975%20393.488C664.975%20393.488%20664.975%20393.824%20664.975%20394.496C664.975%20395.168%20664.942%20395.734%20664.877%20396.193C664.811%20396.652%20664.729%20396.964%20664.631%20397.128C664.532%20397.292%20664.459%20397.439%20664.409%20397.57C664.36%20397.701%20664.319%20397.783%20664.286%20397.816C664.286%20397.833%20664.295%20397.841%20664.311%20397.841C664.377%20397.841%20664.639%20397.759%20665.098%20397.595C666.672%20397.103%20667.459%20396.078%20667.459%20394.521C667.459%20392.93%20666.508%20391.823%20664.606%20391.2L664.114%20391.028L664.213%20391.299C664.426%20391.545%20664.647%20392.078%20664.877%20392.897C664.942%20393.291%20664.975%20393.824%20664.975%20394.496V393.488ZM664.286%20400.718C663.417%20399.505%20662.975%20398.898%20662.958%20398.898C662.86%20398.898%20662.786%20398.915%20662.737%20398.948C662.458%20398.948%20662.204%20398.972%20661.975%20399.021H661.384L665.811%20405.834H668.098L667.779%20405.465C666.91%20404.498%20665.746%20402.915%20664.286%20400.718Z%22%20fill%3D%22%23B8860B%22%2F%3E%3Cpath%20d%3D%22M670.997%20391.547C671.008%20391.512%20671.02%20391.466%20671.032%20391.408C671.043%20391.35%20671.072%20391.223%20671.119%20391.025C671.165%20390.828%20671.223%20390.649%20671.292%20390.486C671.362%20390.324%20671.449%20390.133%20671.553%20389.913C671.658%20389.692%20671.773%20389.519%20671.901%20389.391C672.029%20389.264%20672.179%20389.136%20672.353%20389.008C672.527%20388.881%20672.718%20388.829%20672.927%20388.852C673.263%20388.852%20673.559%20388.922%20673.814%20389.061C674.069%20389.2%20674.248%20389.327%20674.353%20389.443C674.457%20389.559%20674.521%20389.675%20674.544%20389.791C674.567%20389.861%20674.585%20389.895%20674.596%20389.895C674.608%20389.895%20674.654%20389.855%20674.735%20389.774C675.292%20389.171%20675.935%20388.869%20676.665%20388.869H676.717C677.274%20388.869%20677.691%20389.107%20677.969%20389.582C678.062%20389.803%20678.109%20390.017%20678.109%20390.226C678.109%20390.573%20678.033%20390.852%20677.882%20391.06C677.732%20391.269%20677.581%20391.402%20677.43%20391.46C677.28%20391.518%20677.135%20391.541%20676.996%20391.53C676.764%20391.53%20676.578%20391.466%20676.439%20391.338C676.3%20391.211%20676.231%20391.043%20676.231%20390.834C676.231%20390.278%20676.555%20389.907%20677.204%20389.721C676.949%20389.571%20676.741%20389.495%20676.578%20389.495C675.929%20389.495%20675.332%20389.924%20674.787%20390.782C674.671%20390.968%20674.585%20391.159%20674.527%20391.356C674.469%20391.553%20674.272%20392.324%20673.935%20393.668C673.541%20395.291%20673.315%20396.155%20673.257%20396.259C673.107%20396.572%20672.857%20396.729%20672.51%20396.729C672.359%20396.729%20672.237%20396.694%20672.144%20396.624C672.052%20396.555%20671.988%20396.485%20671.953%20396.416C671.918%20396.346%20671.907%20396.288%20671.918%20396.242C671.918%20396.091%20672.15%20395.094%20672.614%20393.251C673.078%20391.408%20673.309%20390.382%20673.309%20390.173C673.309%20389.721%20673.159%20389.495%20672.857%20389.495C672.614%20389.495%20672.405%20389.658%20672.231%20389.982C672.057%20390.307%20671.918%20390.66%20671.814%20391.043C671.71%20391.425%20671.646%20391.634%20671.623%20391.669C671.6%20391.692%20671.507%20391.704%20671.345%20391.704H671.101C671.032%20391.634%20670.997%20391.582%20670.997%20391.547Z%22%20fill%3D%22%23B8860B%22%2F%3E%3Cpath%20d%3D%22M689.428%20396.033C689.428%20396.265%20689.306%20396.381%20689.063%20396.381C689.005%20396.381%20688.877%20396.288%20688.68%20396.103C688.483%20395.917%20687.939%20395.378%20687.046%20394.486L685.238%20392.677L683.412%20394.503C682.183%20395.709%20681.551%20396.323%20681.517%20396.346C681.47%20396.369%20681.43%20396.381%20681.395%20396.381C681.302%20396.381%20681.221%20396.346%20681.151%20396.277C681.082%20396.207%20681.041%20396.126%20681.03%20396.033C681.03%20395.952%20681.111%20395.824%20681.273%20395.651C681.435%20395.477%20681.98%20394.926%20682.908%20393.999L684.716%20392.19L682.908%20390.382C681.969%20389.443%20681.424%20388.892%20681.273%20388.73C681.122%20388.568%20681.041%20388.44%20681.03%20388.348C681.03%20388.243%20681.064%20388.162%20681.134%20388.104C681.204%20388.046%20681.291%20388.017%20681.395%20388.017C681.453%20388.017%20681.493%20388.023%20681.517%20388.035C681.551%20388.058%20682.183%20388.672%20683.412%20389.878L685.238%20391.703L687.046%20389.895C687.939%20389.003%20688.483%20388.464%20688.68%20388.278C688.877%20388.093%20689.005%20388%20689.063%20388C689.306%20388%20689.428%20388.116%20689.428%20388.348C689.428%20388.429%20689.37%20388.533%20689.254%20388.661C689.138%20388.788%20688.57%20389.356%20687.55%20390.365L685.742%20392.19L687.55%20394.016C688.524%20394.99%20689.08%20395.552%20689.219%20395.703C689.359%20395.853%20689.428%20395.964%20689.428%20396.033Z%22%20fill%3D%22%23B8860B%22%2F%3E%3Cpath%20d%3D%22M692.575%20393.807C692.575%20392.637%20693.016%20391.524%20693.897%20390.469C694.778%20389.414%20695.769%20388.881%20696.87%20388.869C697.566%20388.869%20698.099%20389.171%20698.47%20389.773C698.481%20389.762%20698.51%20389.727%20698.557%20389.669C698.603%20389.611%20698.667%20389.547%20698.748%20389.478C698.829%20389.408%20698.922%20389.339%20699.026%20389.269C699.42%20388.991%20699.687%20388.852%20699.826%20388.852C699.872%20388.852%20699.913%20388.869%20699.948%20388.904C699.983%20388.939%20700%20388.968%20700%20388.991C700%20389.095%20699.6%20390.747%20698.8%20393.947C697.977%20397.239%20697.566%20398.908%20697.566%20398.954C697.566%20399.047%20697.826%20399.099%20698.348%20399.111H698.783C698.852%20399.204%20698.887%20399.256%20698.887%20399.267C698.887%20399.279%20698.87%20399.383%20698.835%20399.58C698.789%20399.731%20698.731%20399.841%20698.661%20399.911H698.435C698.076%20399.887%20697.45%20399.876%20696.557%20399.876C696.233%20399.876%20695.937%20399.876%20695.67%20399.876C695.404%20399.876%20695.189%20399.876%20695.027%20399.876C694.865%20399.876%20694.766%20399.882%20694.731%20399.893C694.569%20399.893%20694.488%20399.847%20694.488%20399.754C694.488%20399.719%20694.5%20399.632%20694.523%20399.493C694.569%20399.308%20694.61%20399.198%20694.644%20399.163C694.679%20399.128%20694.795%20399.111%20694.992%20399.111C695.549%20399.111%20695.885%20399.07%20696.001%20398.989C696.093%20398.931%20696.256%20398.421%20696.488%20397.459C696.719%20396.497%20696.841%20396.004%20696.853%20395.981C696.853%20395.969%20696.812%20395.987%20696.731%20396.033C696.175%20396.485%20695.595%20396.711%20694.992%20396.711C694.355%20396.711%20693.792%20396.479%20693.306%20396.016C692.819%20395.552%20692.575%20394.816%20692.575%20393.807ZM698.122%20390.869C697.855%20389.953%20697.421%20389.495%20696.818%20389.495C696.412%20389.495%20696.024%20389.675%20695.653%20390.034C695.282%20390.394%20694.992%20390.863%20694.784%20391.443C694.447%20392.358%20694.21%20393.309%20694.071%20394.294C694.071%20394.329%20694.071%20394.393%20694.071%20394.486C694.071%20394.578%20694.065%20394.648%20694.053%20394.694C694.053%20395.216%20694.158%20395.575%20694.366%20395.772C694.575%20395.969%20694.824%20396.074%20695.114%20396.085C695.74%20396.085%20696.383%20395.668%20697.044%20394.833L697.183%20394.642L698.122%20390.869Z%22%20fill%3D%22%23B8860B%22%2F%3E%3C%2Fg%3E%3Cdefs%3E%3CclipPath%20id%3D%22clip0_1_194%22%3E%3Crect%20width%3D%221089%22%20height%3D%22535%22%20fill%3D%22white%22%20transform%3D%22matrix%281%200%200%20-1%200%20535%29%22%2F%3E%3C%2FclipPath%3E%3C%2Fdefs%3E%3C%2Fsvg%3E)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e6628f19-3f7b-463e-8c74-928f676eb61d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-09-24 20:27:11.216875: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
      "2024-09-24 20:27:11.216954: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
      "2024-09-24 20:27:11.217655: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2024-09-24 20:27:11.221928: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: SSE4.1 SSE4.2 AVX AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    }
   ],
   "source": [
    "# Import libraries\n",
    "import torch\n",
    "from datasets import load_dataset\n",
    "from peft import LoraConfig, get_peft_model\n",
    "from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, pipeline, TrainingArguments\n",
    "from trl import SFTTrainer\n",
    "from textwrap import dedent  # Remove leading whitespace from multiline strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9e339c81-3c1f-4318-b9de-66d971d73e47",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "55dab162c5e941c0b39acbe6737b1740",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Choose a model and load tokenizer and model (using 4bit quantization):\n",
    "model_name = \"microsoft/Phi-3-mini-4k-instruct\"\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "tokenizer.padding_side = 'right'\n",
    "model = AutoModelForCausalLM.from_pretrained(\n",
    "    model_name,\n",
    "    quantization_config=BitsAndBytesConfig(\n",
    "        load_in_4bit=True,\n",
    "        bnb_4bit_quant_type='nf4',\n",
    "        bnb_4bit_compute_dtype=torch.bfloat16),\n",
    "    device_map='cuda:0',\n",
    "    trust_remote_code=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3f008eec-ff6a-47ba-93d1-e32cf5f6afcf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load dataset:\n",
    "medmcqa_train = load_dataset('medmcqa', split='train')\n",
    "medmcqa_val = load_dataset('medmcqa', split='validation')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "685230b3-331a-4179-8487-538b5ea57809",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['id', 'question', 'opa', 'opb', 'opc', 'opd', 'cop', 'choice_type', 'exp', 'subject_name', 'topic_name'],\n",
       "    num_rows: 182822\n",
       "})"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "medmcqa_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7ee4448f-611f-4dd5-b040-ca64a421afe1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': 'e9ad821a-c438-4965-9f77-760819dfa155',\n",
       " 'question': 'Chronic urethral obstruction due to benign prismatic hyperplasia can lead to the following change in kidney parenchyma',\n",
       " 'opa': 'Hyperplasia',\n",
       " 'opb': 'Hyperophy',\n",
       " 'opc': 'Atrophy',\n",
       " 'opd': 'Dyplasia',\n",
       " 'cop': 2,\n",
       " 'choice_type': 'single',\n",
       " 'exp': 'Chronic urethral obstruction because of urinary calculi, prostatic hyperophy, tumors, normal pregnancy, tumors, uterine prolapse or functional disorders cause hydronephrosis which by definition is used to describe dilatation of renal pelvis and calculus associated with progressive atrophy of the kidney due to obstruction to the outflow of urine Refer Robbins 7yh/9,1012,9/e. P950',\n",
       " 'subject_name': 'Anatomy',\n",
       " 'topic_name': 'Urinary tract'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "medmcqa_train[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "16f79a8c-f76d-4003-a66c-23a77ec87dea",
   "metadata": {},
   "outputs": [],
   "source": [
    "# MedMCQA dataset functions:\n",
    "\n",
    "def medmcqa_get_correct_answer(entry):\n",
    "    # entry['cop'] is an integer in the range 0..3 that\n",
    "    # denotes the correct option (a, b, c or d).\n",
    "    options = {0:'opa', 1:'opb', 2:'opc', 3:'opd'}\n",
    "    correct_option = options[entry['cop']]\n",
    "    answer = entry[correct_option]\n",
    "    return answer\n",
    "\n",
    "def medmcqa_get_prompt(entry, include_answer):\n",
    "    options = [\n",
    "        entry[\"opa\"],\n",
    "        entry[\"opb\"],\n",
    "        entry[\"opc\"],\n",
    "        entry[\"opd\"]\n",
    "    ]\n",
    "    messages = [\n",
    "        {'role': 'user', 'content': dedent(f'''\\\n",
    "            You are a medical student taking a multiple-choice exam. Four options are provided for each question. Only one of these options is the correct answer.\n",
    "            Question: {entry[\"question\"]}\n",
    "            Options:\n",
    "            1. {options[0]}\n",
    "            2. {options[1]}\n",
    "            3. {options[2]}\n",
    "            4. {options[3]}\n",
    "            Solve this multiple-choice exam question and provide the correct answer.''')\n",
    "        }\n",
    "    ]\n",
    "    if include_answer:\n",
    "        answer = medmcqa_get_correct_answer(entry)\n",
    "        messages.append(\n",
    "            {'role': 'assistant', 'content': f'Answer: {answer}'}\n",
    "        )\n",
    "    return messages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0cd52677-ee9a-4418-b429-6008daf40b1c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'role': 'user',\n",
       "  'content': 'You are a medical student taking a multiple-choice exam. Four options are provided for each question. Only one of these options is the correct answer.\\nQuestion: Chronic urethral obstruction due to benign prismatic hyperplasia can lead to the following change in kidney parenchyma\\nOptions:\\n1. Hyperplasia\\n2. Hyperophy\\n3. Atrophy\\n4. Dyplasia\\nSolve this multiple-choice exam question and provide the correct answer.'},\n",
       " {'role': 'assistant', 'content': 'Answer: Atrophy'}]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "medmcqa_get_prompt(medmcqa_train[0], include_answer=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "422f273f-b4ff-454d-b9c9-1779525753d2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<|user|>\n",
      "You are a medical student taking a multiple-choice exam. Four options are provided for each question. Only one of these options is the correct answer.\n",
      "Question: Chronic urethral obstruction due to benign prismatic hyperplasia can lead to the following change in kidney parenchyma\n",
      "Options:\n",
      "1. Hyperplasia\n",
      "2. Hyperophy\n",
      "3. Atrophy\n",
      "4. Dyplasia\n",
      "Solve this multiple-choice exam question and provide the correct answer.<|end|>\n",
      "<|assistant|>\n",
      "Answer: Atrophy<|end|>\n",
      "<|endoftext|>\n"
     ]
    }
   ],
   "source": [
    "print(tokenizer.apply_chat_template(medmcqa_get_prompt(medmcqa_train[0], include_answer=True), tokenize=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "02106dac-cf88-4369-a567-58971b26e8e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "medmcqa_train = medmcqa_train.map(lambda entry: {'messages': medmcqa_get_prompt(entry, include_answer=True)})\n",
    "medmcqa_val = medmcqa_val.map(lambda entry: {'messages': medmcqa_get_prompt(entry, include_answer=False)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "42705167-5120-403e-b51c-fedf70923500",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': 'e9ad821a-c438-4965-9f77-760819dfa155',\n",
       " 'question': 'Chronic urethral obstruction due to benign prismatic hyperplasia can lead to the following change in kidney parenchyma',\n",
       " 'opa': 'Hyperplasia',\n",
       " 'opb': 'Hyperophy',\n",
       " 'opc': 'Atrophy',\n",
       " 'opd': 'Dyplasia',\n",
       " 'cop': 2,\n",
       " 'choice_type': 'single',\n",
       " 'exp': 'Chronic urethral obstruction because of urinary calculi, prostatic hyperophy, tumors, normal pregnancy, tumors, uterine prolapse or functional disorders cause hydronephrosis which by definition is used to describe dilatation of renal pelvis and calculus associated with progressive atrophy of the kidney due to obstruction to the outflow of urine Refer Robbins 7yh/9,1012,9/e. P950',\n",
       " 'subject_name': 'Anatomy',\n",
       " 'topic_name': 'Urinary tract',\n",
       " 'messages': [{'content': 'You are a medical student taking a multiple-choice exam. Four options are provided for each question. Only one of these options is the correct answer.\\nQuestion: Chronic urethral obstruction due to benign prismatic hyperplasia can lead to the following change in kidney parenchyma\\nOptions:\\n1. Hyperplasia\\n2. Hyperophy\\n3. Atrophy\\n4. Dyplasia\\nSolve this multiple-choice exam question and provide the correct answer.',\n",
       "   'role': 'user'},\n",
       "  {'content': 'Answer: Atrophy', 'role': 'assistant'}]}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "medmcqa_train[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "00a74a47-d857-41ed-8df3-f04719a5ade7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': '45258d3d-b974-44dd-a161-c3fccbdadd88',\n",
       " 'question': 'Which of the following is not true for myelinated nerve fibers:',\n",
       " 'opa': 'Impulse through myelinated fibers is slower than non-myelinated fibers',\n",
       " 'opb': 'Membrane currents are generated at nodes of Ranvier',\n",
       " 'opc': 'Saltatory conduction of impulses is seen',\n",
       " 'opd': 'Local anesthesia is effective only when the nerve is not covered by myelin sheath',\n",
       " 'cop': 0,\n",
       " 'choice_type': 'multi',\n",
       " 'exp': None,\n",
       " 'subject_name': 'Physiology',\n",
       " 'topic_name': None,\n",
       " 'messages': [{'content': 'You are a medical student taking a multiple-choice exam. Four options are provided for each question. Only one of these options is the correct answer.\\nQuestion: Which of the following is not true for myelinated nerve fibers:\\nOptions:\\n1. Impulse through myelinated fibers is slower than non-myelinated fibers\\n2. Membrane currents are generated at nodes of Ranvier\\n3. Saltatory conduction of impulses is seen\\n4. Local anesthesia is effective only when the nerve is not covered by myelin sheath\\nSolve this multiple-choice exam question and provide the correct answer.',\n",
       "   'role': 'user'}]}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "medmcqa_val[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70fd8bf0-709f-4c4e-9579-76ea62a844f3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "db8cfc1c-3d27-4211-b4e5-1c826ca128a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = pipeline(\"text-generation\", model=model, tokenizer=tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "80d4589d-2ea7-419b-8e62-1f02228bdc78",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You are not running the flash-attention implementation, expect numerical differences.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.52 s, sys: 42.2 ms, total: 2.56 s\n",
      "Wall time: 2.55 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "with torch.cuda.amp.autocast():\n",
    "    output = pipe(medmcqa_val[0]['messages'], max_new_tokens=256, return_full_text=False, do_sample=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "48a3a0c5-084c-4fa1-927b-a7956a1dff80",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " The correct answer is option 1: Impulse through myelinated fibers is slower than non-myelinated fibers. Myelinated nerve fibers actually conduct impulses faster than non-myelinated fibers due to saltatory conduction.\n"
     ]
    }
   ],
   "source": [
    "print(output[0]['generated_text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55f10233-243b-495c-bf90-29fa5a82d80f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "dad86132-cba5-4126-8fad-f5f239459bd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add low-rank adapters (LORA) to the model:\n",
    "peft_config = LoraConfig(\n",
    "    task_type='CAUSAL_LM',\n",
    "    r=16,\n",
    "    lora_alpha=32,  # thumb rule: lora_alpha should be 2*r\n",
    "    lora_dropout=0.05,\n",
    "    bias='none',\n",
    "    target_modules='all-linear',\n",
    ")\n",
    "model = get_peft_model(model, peft_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "143cdbed-272d-437e-93a7-a04caebd1364",
   "metadata": {},
   "outputs": [],
   "source": [
    "training_arguments = TrainingArguments(\n",
    "    output_dir='phi-3-mini-instruct-medmcqa',\n",
    "    per_device_train_batch_size=8,\n",
    "    gradient_accumulation_steps=1,\n",
    "    gradient_checkpointing=True, # Gradient checkpointing improves memory efficiency, but slows down training,\n",
    "        # e.g. Mistral 7B with PEFT using bitsandbytes:\n",
    "        # - enabled: 11 GB GPU RAM and 12 samples/second\n",
    "        # - disabled: 40 GB GPU RAM and 8 samples/second\n",
    "    gradient_checkpointing_kwargs={'use_reentrant': False},  # Use newer implementation that will become the default.\n",
    "    optim='adamw_torch',\n",
    "    learning_rate=2e-4,  # QLoRA suggestions: 2e-4 for 7B or 13B, 1e-4 for 33B or 65B\n",
    "    logging_strategy='steps',  # 'no', 'epoch' or 'steps'\n",
    "    logging_steps=10,\n",
    "    save_strategy='no',  # 'no', 'epoch' or 'steps'\n",
    "    # save_steps=2000,\n",
    "    # num_train_epochs=5,\n",
    "    max_steps=100,\n",
    "    bf16=True,  # mixed precision training\n",
    "    report_to='none',  # disable wandb\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "39712d55-0eee-415b-a096-2d57703c8545",
   "metadata": {},
   "outputs": [],
   "source": [
    "def formatting_func(entries):\n",
    "    output_texts = []\n",
    "    for i in range(len(entries)):\n",
    "        text = tokenizer.apply_chat_template(entries['messages'][i], tokenize=False)\n",
    "        output_texts.append(text)\n",
    "    return output_texts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "5ce24824-364d-4123-91c7-be93f7458527",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7f05390c9cf149788ca85b3a73a7b4c3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/182822 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/gpfs/opt/sw/jupyterhub/envs/conda/vsc5/jupyterhub-huggingface-v2/lib/python3.10/site-packages/accelerate/accelerator.py:436: FutureWarning: Passing the following arguments to `Accelerator` is deprecated and will be removed in version 1.0 of Accelerate: dict_keys(['dispatch_batches', 'split_batches', 'even_batches', 'use_seedable_sampler']). Please pass an `accelerate.DataLoaderConfiguration` instead: \n",
      "dataloader_config = DataLoaderConfiguration(dispatch_batches=None, split_batches=False, even_batches=True, use_seedable_sampler=True)\n",
      "  warnings.warn(\n",
      "Detected kernel version 4.18.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.\n"
     ]
    }
   ],
   "source": [
    "trainer = SFTTrainer(\n",
    "    model=model,\n",
    "    args=training_arguments,\n",
    "    train_dataset=medmcqa_train,\n",
    "    tokenizer=tokenizer,\n",
    "    formatting_func=formatting_func,\n",
    "    max_seq_length=1024,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "8b3be793-debb-4035-9b31-15f3c8d24c02",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='100' max='100' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [100/100 00:57, Epoch 0/1]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.137900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>0.605000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30</td>\n",
       "      <td>0.640300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>40</td>\n",
       "      <td>0.624500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>0.588500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>60</td>\n",
       "      <td>0.620700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>70</td>\n",
       "      <td>0.548200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>80</td>\n",
       "      <td>0.594400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>90</td>\n",
       "      <td>0.577900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>100</td>\n",
       "      <td>0.586800</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "result = trainer.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "bfc5501d-51a5-450a-935b-7d083646abf6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=100, training_loss=0.6524051284790039, metrics={'train_runtime': 58.5258, 'train_samples_per_second': 13.669, 'train_steps_per_second': 1.709, 'total_flos': 3138931125927936.0, 'train_loss': 0.6524051284790039, 'epoch': 0.36})"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "8ae87e0b-e067-49ff-8931-44ade8c03f99",
   "metadata": {},
   "outputs": [],
   "source": [
    "# trainer.save_model()\n",
    "\n",
    "# model.save_pretrained(\"output_dir\")\n",
    "# tokenizer.save_pretrained(\"output_dir\")\n",
    "\n",
    "# model.push_to_hub(\"your-name/model-name\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "9b24ab3e-a1b7-48d8-a8f0-ca3df7219310",
   "metadata": {},
   "outputs": [],
   "source": [
    "## To load fine-tuned model, use:\n",
    "\n",
    "# from peft import AutoPeftModelForCausalLM\n",
    "# from transformers import AutoTokenizer\n",
    "# model_name = 'phi-3.5-mini-instruct-medmcqa'\n",
    "# model = AutoPeftModelForCausalLM.from_pretrained(\n",
    "#     model_name,\n",
    "#     torch_dtype=torch.bfloat16,\n",
    "#     device_map='cuda'\n",
    "# )\n",
    "# tokenizer = AutoTokenizer.from_pretrained(model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "f35eae93-28eb-4256-ad37-6ed9c874f911",
   "metadata": {},
   "outputs": [],
   "source": [
    "# model = AutoModelForCausalLM.from_pretrained(base_model_id)\n",
    "# model.load_adapter(peft_model_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "2d21fc9d-dc3a-4757-a1ba-534f9d9b8072",
   "metadata": {},
   "outputs": [],
   "source": [
    "# model.enable_adapters()\n",
    "\n",
    "# model.disable_adapters()\n",
    "\n",
    "# model.merge_and_unload()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be0397a9-ba92-4515-82b1-9b4925f6ed98",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "d6729721-4214-40d5-97bc-60e2f463ac74",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.68 s, sys: 6.66 ms, total: 1.69 s\n",
      "Wall time: 1.69 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "with torch.cuda.amp.autocast():\n",
    "    output = pipe(medmcqa_val[0]['messages'], max_new_tokens=256, return_full_text=False, do_sample=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "61b316e5-85cd-46dc-9592-b4830eaefeda",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Answer: Impulse through myelinated fibers is slower than non-myelinated fibers\n"
     ]
    }
   ],
   "source": [
    "print(output[0]['generated_text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "30a9c2d7-e497-4b44-846b-9ce400061fb1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Impulse through myelinated fibers is slower than non-myelinated fibers'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "medmcqa_get_correct_answer(medmcqa_val[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "349a7050-cb95-4b4b-8a3c-54e61b0f7a5a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c84df2b8-fef2-4115-ac96-b2bda69a9d15",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "0865ed90-e237-4efb-b193-87904fd62893",
   "metadata": {},
   "source": [
    "## Hands-On Exercise: Finetune a Model on a Chat Dataset\n",
    "\n",
    "For the following hands-on exercise, we are going to train a model on a different dataset. The OpenAssistant Conversations dataset consists of human-generated, human-annotated assistant-style conversations. We will use the dataset `timdettmers/openassistant-guanaco`, which is a small subset of the OpenAssistant Conversations dataset, consisting only of the highest-rated entries of the original dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "242ae8aa-d45b-4924-9f0f-f640824affcd",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n"
     ]
    }
   ],
   "source": [
    "# Load the guanaco dataset\n",
    "guanaco_train = load_dataset('timdettmers/openassistant-guanaco', split='train')\n",
    "guanaco_test = load_dataset('timdettmers/openassistant-guanaco', split='test')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "bc19f917-1cb4-48c8-bf60-6b27e3e63d0e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['text'],\n",
       "    num_rows: 9846\n",
       "})"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "guanaco_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "4f918509-d530-47f7-9633-6d390d5bd16b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['text'],\n",
       "    num_rows: 518\n",
       "})"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "guanaco_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "7d197bc1-c141-45ca-9feb-b8c93ef4242d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': '### Human: Can you write a short introduction about the relevance of the term \"monopsony\" in economics? Please use examples related to potential monopsonies in the labour market and cite relevant research.### Assistant: \"Monopsony\" refers to a market structure where there is only one buyer for a particular good or service. In economics, this term is particularly relevant in the labor market, where a monopsony employer has significant power over the wages and working conditions of their employees. The presence of a monopsony can result in lower wages and reduced employment opportunities for workers, as the employer has little incentive to increase wages or provide better working conditions.\\n\\nRecent research has identified potential monopsonies in industries such as retail and fast food, where a few large companies control a significant portion of the market (Bivens & Mishel, 2013). In these industries, workers often face low wages, limited benefits, and reduced bargaining power, leading to a situation where they are dependent on the employer for their livelihood. This dependence can result in further suppression of wages and a decline in working conditions.\\n\\nOverall, the concept of monopsony is essential to understanding the dynamics of labor markets and the impact of market power on workers. Further research is needed to understand the extent and impact of monopsonies on the economy and to develop policies to address this issue.\\n\\nReferences:\\nBivens, J., & Mishel, L. (2013). The Pay of Corporate Executives and Financial Professionals as Evidence of Rents in Top 1 Percent Incomes. Journal of Economic Perspectives, 27(3), 57-78.### Human: Now explain it to a dog'}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "guanaco_train[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "f5999242-d1f4-41f4-81e7-12e34cf7c898",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': \"### Human: What do you think about ChatGPT?### Assistant: As an open source alternative to ChatGPT, I do not have personal opinions. However, I can provide objective information about ChatGPT's capabilities and limitations based on its architecture and training data.\\n\\nChatGPT is a powerful language model based on the GPT (Generative Pre-trained Transformer) architecture developed by OpenAI. It has been trained on a large corpus of text data and is capable of generating human-like responses to a wide variety of prompts and questions. ChatGPT's ability to generate coherent and contextually relevant responses is based on its ability to understand the structure of language and make predictions about what words and phrases are likely to follow in a given context.\\n\\nHowever, like all language models, ChatGPT has limitations. Its responses are based solely on the patterns it has learned from its training data and it may generate responses that are biased, incomplete, or inaccurate based on the input it receives. Additionally, ChatGPT may struggle with tasks that require a deep understanding of context, such as detecting sarcasm or understanding humor.\\n\\nOverall, ChatGPT is a powerful language model that has demonstrated impressive capabilities in generating human-like responses. However, it is important to recognize its limitations and use it appropriately, while continuing to develop and refine its capabilities through ongoing research and development.### Human: What are your thoughts on the censorship of ChatGPT's output and its liberal biases?### Assistant: As a large language model trained on text from the internet, ChatGPT will have learned its biases according to the opinions of the people who have written the text online. Some of the text it has seen online could potentially contain harmful content that incites violence or aggression towards other people, so ChatGPT is perfectly capable of reproducing these views and insisting on their validity in face of opposition. In order to act as a useful assistant, ChatGPT should be able to distinguish fact from opinion. As it might not be able to do this perfectly, censorship is probably the safest approach OpenAI could take to prevent toxic content from the model output.\"}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "guanaco_test[3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "fffdcf29-2d7f-4634-b615-18c9755a45a4",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f97f6bf780604cf095dbaf6a5e6f783e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Choose a model and load tokenizer and model (using 4bit quantization):\n",
    "model_name = \"microsoft/Phi-3-mini-4k-instruct\"\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "tokenizer.padding_side = 'right'\n",
    "model = AutoModelForCausalLM.from_pretrained(\n",
    "    model_name,\n",
    "    quantization_config=BitsAndBytesConfig(\n",
    "        load_in_4bit=True,\n",
    "        bnb_4bit_quant_type='nf4',\n",
    "        bnb_4bit_compute_dtype=torch.bfloat16),\n",
    "    device_map='cuda:0',\n",
    "    trust_remote_code=True\n",
    ")\n",
    "pipe = pipeline(\"text-generation\", model=model, tokenizer=tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "622ba4c3-abe1-4b56-93fe-e62d81a8b708",
   "metadata": {},
   "outputs": [],
   "source": [
    "peft_config = LoraConfig(\n",
    "    task_type='CAUSAL_LM',\n",
    "    r=16,\n",
    "    lora_alpha=32,  # thumb rule: lora_alpha should be 2*r\n",
    "    bias='none',\n",
    "    target_modules='all-linear',\n",
    ")\n",
    "model = get_peft_model(model, peft_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "95d07724-cf03-4ecd-a514-5203139da021",
   "metadata": {},
   "outputs": [],
   "source": [
    "training_arguments = TrainingArguments(\n",
    "    # When using newer versions of `trl`, use SFTConfig(...) instead of TrainingArguments(...).\n",
    "    output_dir='phi-3-mini-instruct-guanaco',\n",
    "    per_device_train_batch_size=8,\n",
    "    gradient_accumulation_steps=1,\n",
    "    gradient_checkpointing=True, # Gradient checkpointing improves memory efficiency, but slows down training,\n",
    "        # e.g. Mistral 7B with PEFT using bitsandbytes:\n",
    "        # - enabled: 11 GB GPU RAM and 12 samples/second\n",
    "        # - disabled: 40 GB GPU RAM and 8 samples/second\n",
    "    gradient_checkpointing_kwargs={'use_reentrant': False},  # Use newer implementation that will become the default.\n",
    "    optim='adamw_torch',\n",
    "    learning_rate=2e-4,  # QLoRA suggestions: 2e-4 for 7B or 13B, 1e-4 for 33B or 65B\n",
    "    logging_strategy='steps',  # 'no', 'epoch' or 'steps'\n",
    "    logging_steps=10,\n",
    "    save_strategy='no',  # 'no', 'epoch' or 'steps'\n",
    "    # eval_strategy='steps',\n",
    "    # eval_steps=20,\n",
    "    # save_steps=2000,\n",
    "    # num_train_epochs=5,\n",
    "    max_steps=100,\n",
    "    bf16=True,  # mixed precision training: faster, but uses more memory\n",
    "    report_to='none',  # disable wandb\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "c646cc8c-3b5a-4e1c-8d5a-7907767b92ac",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5a78ae7cf3f944f8b732f036348153dd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/9846 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "139657c24d804f0586ff2a54a4cd6196",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/518 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/gpfs/opt/sw/jupyterhub/envs/conda/vsc5/jupyterhub-huggingface-v2/lib/python3.10/site-packages/accelerate/accelerator.py:436: FutureWarning: Passing the following arguments to `Accelerator` is deprecated and will be removed in version 1.0 of Accelerate: dict_keys(['dispatch_batches', 'split_batches', 'even_batches', 'use_seedable_sampler']). Please pass an `accelerate.DataLoaderConfiguration` instead: \n",
      "dataloader_config = DataLoaderConfiguration(dispatch_batches=None, split_batches=False, even_batches=True, use_seedable_sampler=True)\n",
      "  warnings.warn(\n",
      "Detected kernel version 4.18.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.\n"
     ]
    }
   ],
   "source": [
    "trainer = SFTTrainer(\n",
    "# Excercise: Fill in this cell so that the `trainer` can then be used for training and evaluation.\n",
    "# Compared to the first example in this notebook, there are two additional parameters needed\n",
    "# and one parameter specified above is not needed here. You can have a look at the documentation\n",
    "# of SFTTrainer.\n",
    "# Hint 1: We now want to also pass the test dataset to `trainer`, so that we can use `trainer.evaluate()`\n",
    "# before and after finetuning and compare the results.\n",
    "# Hint 2: The part of the training dataset that will be used for training is not in the chat messages\n",
    "# format, but already available as text strings.\n",
    "\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "a53dde88-8f09-4ada-9a2b-e5f08b8b8f34",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='130' max='65' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [65/65 06:13]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 28.1 s, sys: 16.6 s, total: 44.8 s\n",
      "Wall time: 52 s\n",
      "Loss on test dataset before training: 1.408\n"
     ]
    }
   ],
   "source": [
    "%time eval_result = trainer.evaluate()\n",
    "print(f\"Loss on test dataset before training: {eval_result['eval_loss']:.3f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "4b85c3fd-1c4b-45e8-82ed-6f4ec27f59aa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='100' max='100' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [100/100 04:23, Epoch 0/1]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.234800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>1.155200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30</td>\n",
       "      <td>1.157000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>40</td>\n",
       "      <td>1.155500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>1.178800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>60</td>\n",
       "      <td>1.252000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>70</td>\n",
       "      <td>1.212700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>80</td>\n",
       "      <td>1.134400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>90</td>\n",
       "      <td>1.108000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>100</td>\n",
       "      <td>1.121900</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_result = trainer.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "fdfa67fd-e53d-4ea0-a898-a8372d1b1169",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=100, training_loss=1.171025915145874, metrics={'train_runtime': 267.1724, 'train_samples_per_second': 2.994, 'train_steps_per_second': 0.374, 'total_flos': 1.5497134489387008e+16, 'train_loss': 1.171025915145874, 'epoch': 0.08})"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "a4a2fd2d-d2e0-4847-8a6e-bec7abc50d6e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 28.1 s, sys: 16.8 s, total: 44.9 s\n",
      "Wall time: 54.9 s\n",
      "Loss on test dataset after training: 1.228\n"
     ]
    }
   ],
   "source": [
    "%time eval_result = trainer.evaluate()\n",
    "print(f\"Loss on test dataset after training: {eval_result['eval_loss']:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a535291-da18-464c-af18-bdc48185173a",
   "metadata": {},
   "source": [
    "**Note:** Compare the loss on the test dataset before and after training."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a40ccea-a8d2-4a4c-940a-93bfc67e35aa",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Solution:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "986f6c92-288e-41f2-95e4-444ddadc497c",
   "metadata": {},
   "source": [
    "``` python\n",
    "trainer = SFTTrainer(\n",
    "    model=model,\n",
    "    args=training_arguments,\n",
    "    train_dataset=guanaco_train,\n",
    "    eval_dataset=guanaco_test,\n",
    "    tokenizer=tokenizer,\n",
    "    # formatting_func=formatting_func,  # We don't need a formatting function for the guanaco dataset.\n",
    "    dataset_text_field='text',\n",
    "    max_seq_length=1024,\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "661ab480-5d07-46b3-9493-dcf5c2d06e34",
   "metadata": {},
   "source": [
    "### (End of solution)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "181e9b56-528e-4e5c-aab5-9b06a5995c87",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5e631ca-477f-43e0-92c5-16662ca4e968",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6090d0d-c85a-4ccb-a3b2-67b12f38266f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "ffc979e7-7ec4-4758-882c-b7d2dee9f2dc",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "In this notebook, we explored Parameter-Efficient Fine-Tuning (PEFT) techniques using adapter layers to efficiently fine-tune pre-trained models. You learned how to configure and apply adapters, perform fine-tuning, and evaluate the model's performance.\n",
    "\n",
    "### Next Steps:\n",
    "In the next notebook, we will explore a technique to speed up model training and inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "18bbbfb9-323c-485f-ac4f-77d735c06adf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'status': 'ok', 'restart': False}"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Shut down the kernel to release memory\n",
    "import IPython\n",
    "\n",
    "app = IPython.Application.instance()\n",
    "app.kernel.do_shutdown(restart=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bdc04584-0b52-403f-8fc6-2dae3fe8bc56",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}