From 033b5f98a4fed8cad1c53df630d9ca76035c72f8 Mon Sep 17 00:00:00 2001 From: julianaklulo Date: Tue, 28 Nov 2023 14:36:21 +0000 Subject: [PATCH] deploy: 38d6e09fa75720b84e2dfdbc4876185dbb83d556 --- objects.inv | Bin 7402 -> 7445 bytes reference/agent/index.html | 171 +++++++++++++++++++++++++++++++++++++ search/search_index.json | 2 +- sitemap.xml.gz | Bin 127 -> 127 bytes 4 files changed, 172 insertions(+), 1 deletion(-) diff --git a/objects.inv b/objects.inv index 7fd9303d5e4a3d88e549ddc7ee76daf716c43ba0..d4c78330cb5c7f78a70bf4df631f422ffa972ba4 100644 GIT binary patch delta 6595 zcmV;!89e6dIh8uF!vcS-8L5FLz!c4yK;JK6yuCS|KaV{pn3m2_{7*?oH!H=MiUXV% z1wef$Al<15?ua7Hejo@#nXO7#cUHQoGv21}G2O}hjaA@VTGwsBg4t@Rhh7?1IJt?^ z3p95M`?wSx14G~@z}!{9Ini*7mIS*tDqtsGS46o-lhj?8kM@6UX23+OMae}jZ2-v6 zX?ZSC>+p~y<&pD1|7qx2DoiIb+F-Z~3tMpAiNV%Rq!fEg%jUvd;s&EY!Q;tgjOHlb z6s!vvM@v8_9yP8qM0l~9hi>^8V(VdGc4l~v}rS;c=v5W%pLy_2+|Zk zion!nAcSH*w8q-!ca$|0j;cc_A_`jNKZgR^oF_kAIr9Yf^5r}k%|$<<`oYyjbIrrY zZ(E#^NzZ>t`*S>*E!T>f0T--?!2}!z^bfh(g!E(`-YH44v?!7!Ld#l983Iz<6WrTQ z)u)^!@6~JipOW5Yjn<>hJMqIE#UO&a_#{Zmv{ayJ%r&H&n=n~duPZ_m(Mc1S%4=iA zV2EFsDPQ3a5QQj_A!dNFl&uG>E*$1r8+}e}ovVMoePR+xKoNT~np0t03YrPzjJR*W zzL%pQ665lxd)%BK5q`}xL?XF}IzeG6l0tr+1F1lUm>xMX#h_vg4V55R3a>u(_@DaifSkaQr;~$E*RT_BYw-K zR0)3yA-V}kPlC3ceXJD7i7XF{0_`h$%NTJ z%p{lxo%Jcx2hV zU))%cE;n^39w>XB8SN`vdH+O!F9fXC3HT|05p(5hTI8%H`B9q$a0XM0l*0IJzsXNh zl`9DR!K0Ml5vdyN?F*_gLP%I$Fqq2PeUwN!qlE`%-tU~;@RVI8L9a&`bj1#{Z=*QS&0qHnu}SD_5>V% zmUojEXWeP-KTloxbQd{`cc>+u6gRJxe+a1o1SN5vvo$Em5(I}FM^%|IhO5~DSI|9N66B%aojO6&qnP~xp-Ex z%P>)R8%wYdPEZF(EZ7i#hUahMy4XphMA@HXdZKL=_t8c(90P-pJgS{KAu3GCRn$2~ zhM{>_b9zFx*JZ%0tI{Kk2BVLED8OPlU3gEv z4Yz$Pr9alevk5-xDleh-Zq@Is?328xdGm#)v>}mPMKCvbH$Jq@I1Yd~m_QATCE8f`(#d zkL^Jd*sV`<*y-$lcA~>Z4eBVAtDNW~Q8q!4?=e_qft_};NZQxx7-7`7s+onEBf?vf zAPlhxEI?0p_}bi9KY0+p-Zz9+x6ki61(+|~g`JR;7~iH#p{xlW`aj0LPB>&5jW9!* zJqmN|Mu|kFX?T_v$NO+nML2VXfxcF6L`u8N| zt4;hdAiJRG19o;1(RqJ%5pgHTobUldof;wQw>FXeDiwy`7Lw|AmV*^lRy4nSj8%sb zkNTst*HCBEa9$(=EYDJ2#Dwfeakm3;%asPg7cqH%4DsH)I@SP+dyHExOx*UmQX;_b zfo&O^PXsFj;((&FyRZ`eVO$9ZOCL@JTbx?dh+Y5WqN{@dr_N0eFOKEM9$-{pZFtsD zC4JAt5-Xp`ylhweJq`srb2~!Lvpd;CBow>Tg}Y1NS>~qa?c7;yd*;rW*T&cDJM%ig zNj+zOuA80Kb7r*Ri9L5tM>)0U%y^@dd(M1zKE3D6bDx}bXL?6Tuy!ZB*-5_eJWFYs z-r?*yf1TfEA#86H$urCz-Go#zYC%z8#xWbPHXujOf8T zw2B;#@QQ1Kp6)mi!AryO!U!hD3k|{)?syRaJh<+_KMeZx@H+^m{=lg}yjY?6)c1gY zD_X_$h$#|OZ&;v48@M`X029py9b>8Kuz?IVEj)zXX5KJ~f`RO32Z`IOIqe;uce~OK zt`U~O_DH;lj5AY68V=AHn2v+KB1lz+r=&RtS>ezpq1HGkLzFZQc=NEPI)1l*^wJbp zj8nIYQ_K>itRv7`WM4Sez}GG?Kmz7}og4_(&};oAh@c3FH9SRv>KO}EIBPiWGCk#G zJAjowz+HEGJ_j%r4|9W^fzc6+*WK==Gu;{oH$3$B$Hn2s+mFt-w#7j!E?SEN+|M-g zxezU`jyqpqX5w@RV-)IIGRs|8aX3(`Z99Sz@@L_tbV4hfUeUO?MNk{K3dz@h>10h2 zY~m_RqlAshFGR61dC9My;rX`E>1%cUie6FixokY|sE`1Uh|;{glBmM0;cnF{N}h_e zd`VFG>fMbbVD^l%m!zyplvFrLi-K=0k_5enxXjD5Ql2_i9Jcz*|MV>_S)+OmJ_{J@ zYiv#tUjnFT3;oCFF9Fkq-&r$%U(yWRw>@}~({oy$c|@7@v#v_s!*PzGk%ZEMvKfN+ z$)M_;iu{5|ot9yx<@J(W%ji8rwfL!}?r)@wj(7NKU3nnIt0!sBmH=R|g)8IJL=vNr zDPG`0NYV>R=tu3n(K-~|=yE)jzJc>A=n$r*8=90xuzn__ZA@49|?s$2yTG2WGM;yJI-aH1MgJPy{>Kxn*2An})AKyAmj zF2A~=qZQR%vJr5Qu!CrScNx!5NrQ{vDw~WB9~_4<neL0+aD%-?Q3@EVJRS^wr~b8CLa0SOu<58O1Ju?P zRKxN#>#tQ+yW>)SRw5}_k)f;xx_Uw7BMQ(-Us4UstvJj^7yAc6p1?;w^na2_A0ya8{r! zL#7C7Ayv(qnoq~LdNbX%Yh=p4h7n;6kY&tVYU>|{MABD()aj46d${(gG71H* zuCqw^-ewamGS0_3f>Ck{TNGxwBRzrW*+c5|T__M$TcyjO6L9BJJz_W#Dh!n6Nt{q= zBRHMJHp>`(;Hqv#Sc(PKu3o4iPeE@#IHVak7Czm7_YA@@xau1IW*;}w5;EfDX%`9D zHo6>(!v6VLw_;M;9v`Ns;=zv8oQzgh<{ zgl0UMFM?lzV%yU84K4>HQ;5nFYMt=sNPyg`Njuer)_EP5vHg`#yG8!w*)b+%S5o9u zMxSecYrNPgD31P&in_ZBe-Poo2RCKOpG)G8;jBk#BE`aLot{!u*Q>mR+@p`kQf<7A z{o=nQO%RB9+14AwklSS{0>4clP`3$q3sLX^ZvneTR9XOg1dHqjgG=~HO~V`Wq2Qrr?kElRttdW_P435HsXdg8LlSWbHebk#P5ld2}A4FVLi zrdI`Id9?(GeGsS0XmE5wuX!1?4Dg~=L&M5;>xh5JF77})J#p!mjEk!Uh@e2&c8n;$ zjrTO1X!2N!CHgkYrWD3+O8SmS&1MPoxqL-MRU?0SN)(;QNBU79Qzo)SKcn*DPL=|H z>z39D;lGE1t>YO;@SH#J!n1wfgC(>OS_yLO8Q-_GXh1jm{UJnv6C=_Tlte-iHoq50 zf`wukrwuopb_d&gPLLbl;3j&HyjHhW?bwHEZ6#mNeA}LmRR6%y;KO#4pWlSm0o@l3 zVIAzA8v@gGCm$4DR+bNjI_b&>p+9|p;P66^GuD@GPay*1yBNrl4?z9mntTCsDx0FP zTw~?}s(v9UuD(5Hbe)&kn;}~ZN;3xx1!|{4Z;S=eG+SyuNQ+>JPvLeLePNDEcf2TS zh%X~7PNmzBqVK$|2t)!v@LzynHnRP-;yEH>n_o9 zt(|wbv0O>wk2lSk0mGFsuTkuOaq=Q)H=(uiH0bX1x&#)m`mP_lvuDMCRc7tvU9Di7 zOXwF{n!tFI&^wcUET%Z_G(xQmCU+bBkm2HlR_r;e5Hl@QSa@_VR^o7+ zQxBjAkvho&yfw9n6CjerG7aS;^kPgH6dxp*m~y(e%@ zIl@Sj!s-;$JS|oiSL5ndQ|Nwsy5~0tJyq|Z?cRZ|*66(et!jeojzPeG|Hz-@S~C*j zO_c}C@C{bKE#`XamH7V<+6A7o1mc>SU3ox%$}uwBKog6)nq~$G$M1D~uLV@S9B?(w z3h?7A_4ksZt8c!88|G@N?eLGU-6P&VVxbIfj;?Blnk}`%^n)=HsB$M1*HW@x55R(c z`0qiQ0onJS5$4`PdP`n^)1pD(oOQ`beoiY{Xrk}tjz%pH95NC5y?nnh>fS>iRkT;CMIH|6DB5T+wO1mqGPPqDad*j z>r9S5`PsFjEf8Eq@6^esCmNv-L~6usH4o43*$L=}vujacP}b|tb3wKQc#oc?!tv)3 z?gK6%e57wUwC2)(_sC8=!MYG~yU>J8g#zj) zb5v(KmWM2TwF-ZUUt7N=t}mrcbFnoXHcrJ}+)l`&3oFfkqiSq-YU+C!W4hwD{;cL^ zM5xNhu+2)1r$cw57vsP$scho(?6v8%r^CyLjQb@n0pZO z&4h?J3DcJjLS2~YTL*ARdl(-x2D@!c%y?|FIWd=ixyc5F`l%tHUVYeo5B0l0khMPQ z*1?SC$YSe1iQGb1G>~3~7gdn$gc;_>TMD{YX&|e#sr5FKy<4$MCx!%y{6CyL<#juNu+bv=Ix$J1GnO0(`$<4Nyz{=Tc8g=(J!fgi~`hw(; z;Ndxci6|29e4EGk%0L8|BF`lVA!y-V-Yy-1pt$4L388yq+P6*nx2ZRb%a0$#&R+aZ z;VB-|jx2vl66Pud9A#8O1s;o3WAQ-@gjMn`27)CXJMjamu=a{B_L1`Yr@Mwe^Lv=v zc&F*zFD=Aq$BKddmoVvJK$t=^HdUAu>Is;CW`8z0M)Iv~FxWPK&f&#!Z(QS>9R6s= zXF5FLx_8TMZ^nDV%16wDH7Tz74eH4J%@pqeBpzA;7)1gHWGLy8V#bpJHF{7Pu;WLT z5j<>onIXf*m;p0-pc$}ZN1K5D(Y=x5-AzSF)-CIbQ0t=Z0`s4mhh5(KXMg-^PoxZ(9!b7}hYzb^1xEL4j`xMkFc7`>dD(P5TVvkaga!6qbPnYBVkEVpKCm}S>2 z17!Z|M9h|9`_LLK-Y(GILw8il1>cZPPivvC$foVT(3=AOGngnEVcm?%!`L`uz*tLX zj0IrtjN#&~o*7z{?K8##n(xMtk#^7+3S>4I0j@A$N9Z9Y(<{Q=#)fe?{y%&>5e?q8 B!sGw| delta 6575 zcmV;g8BpexI_f#F!vcRi=@u}~)*R2D$DZT6rE?VbPtws%3Ne=20Ov&kP#+3NcPfHA zq6o7e2*OZSqY~Dgm2T#WH>rC}cQSut75J9cbsH~Wrc&ymmxdKiZld%8&7HzN7JbLS z5H|_1HWhGAG~9v}!JdT**ooH_QSMMAb=T#iT?+~PoR;STwO)S#Nm3qO4s)^L;!P2)fZo0YbmGzeBRV?i9UOp8 zE^)ij2jQ7ZJpm!HGw zq>|@`CgPRr#MZ< z_4MvhG$k(~8WQDB%XnDAKL}@HiWYTle{v1zc%e7)=zy<4JFad`F{X?!cAw5}#cS@2hEs7+G(6ZK2 zhJe)e1owZoQ}rn)$$Ryh{->n3S)=u6^G^J5N1k?Y7oP;|n3f7OjsJyoa}y@(>UBkE zB06aTQ+aLN5)APRGvzD%0iqBkGQ8=`d~;C$t;p^q`H51$|CRu?j@I*3d{CBjlUJ$4A8sD^SN<=t}Q zg0U?);1wqMHS_1E)unv6Xs3CTzm* zz&wAIOqdPCOoCZ)CwVXn)&Euap+8Vu0A!{3;U-`ZwB+pxnd*41{VN; zRd==#n8D>CW zxYKVb@tvQ?CH)0*L;qX;o|j=#hvNRC`JxdadMF+y+K^bPlWa{V^)i*$_Zj`%2!9j9 z3;&cPgyLBQ$g8haw_8<;ChxIec(G}#K-?e!V>d>G;FyLdciifrE9dSaHIAGipO+Xf@&8&lI49$9wp7dKX<%S|1M2g;sjM*9j^-h~k0 z3jwQj0)9$=#9aBB7CCE4e$*xboWay0r7(W+Yx0v+qT zGqzZ;^q?a-c9DvBPPj@W7$`wh67FTCl;ZrU0i(Kr1tmQX=Y;d*NTAdf!I*Gv0t%K~ zm2sH2XT~RCOH@O)9SvRnIPVq_iV-lN#Om~#m+bd*0@!{tP*B37g(*Op%0+3F<)|sw zT>fW&=&;P8L!EY!y}y~CQ8`@m2Q7c|UB5_&3?^@Tt90${v$VHQ&fXy>+mwnO(;IrY zUGn0TH1^GfQ&0j;WH#U{@|Ix)n3Imfa-)}+A;e%4lQoC2VF$`?Nu*jTM;kD^C5^fq z6xktCr!V@A|NS`Jb~uVXW&)DvyGMSnBCQvHCRdg$-{J5}kX0}P$&OMG^U8=)@Y?Pg z4kL}|G+c#hoO5pt`Qbay5({21iNFeF`DOp^lk~f1uJ0(tmnVzhFSxpO4GGsAdTe)o zJ0ynBd?n$PpP=;zwfDjjK;{cstOIRJFU?~kv6nSvI3=L(v%HB=Y+W!KAAipg~gAl=$GOsA?&Qu7$gzYJieTmEbUORSh&)cH>c! z8i8^d_ef#1+U{{-3A6nTj18KxWHOL}apBS)9TPYAIkLAyD&{U3m@eiiOe1ffmk5=< zH`=7SWA@xVH|H$xFM0!SoDR5O7T~OZ*k%id@orO~MpR+%mQiQpwRBBp+pBTLj#>C& zGWe`SkCU`#rE1F43-53X*jS$1r}SAFE&b*QN-bC{Cn!JIEafP!RX^3 z3b0sC7v7VzXXYnhR02_(V#YFzvmfV30BppTLXuS_MI|7$DQqlHEJA`n>?Bw$r-NBT z6Ka&Xez!GRR)rkPR$5nUEE!XOXdt4;H8Lk;YxX%^!)+f+>5p~rY=V!v%1fxdTlG6D z`y?-F-h81cZAc`eA%?8N_Mo=?$(J@qC3Z){kmwTLhXw@)(^It(D44l3t%HB6F{1P% zHUnZYq97Er`Vpq3tSyc)sb`-pA6zgnh|5ujprM%AV|&m9cI(p|b~?L%o#?PpgE|W3 zDku6#luZ!idkj`tV5gldlJ>PaMi@1&YGz^Pi13yq2tzCa3((UYzBV`3Paedt_YI-d z?elw10p<&LVJ9Rd#<%HGC~Ja;{*SS*6Aqb1Bg{}{kHQ@LQHeK{S5bJor^~a2^77i5 zf3!+SCvgphw?c z{@Lg{p8&KmjXeoyBkFo0(8e<+&DO@zCCApr?=a?;lHKaVCuZb-Y;79$Sx@QfSp}Lbl#s`MBE87 zCw#zAr$)&7txaUVN`>LKg`~QjVg* z>`wL&3B~So;qKCRmbvM9J9k#wp1E`8wej`(&b$tAQqP%x>t?6*oEdF+V$Yq^QBLhS zGv4Uro-?1FPwzSN+$U$-yDnH zv2QA_Fpp_}liy=!i$56WNQl3_l?X>-@FzwIO>cE5PCeZn_g(FDTjc&kBKBdrK-=%v zvcl+V`uC6jM3f@@NhYj?F;T*`Z^z~o-9i^EBYN--ts;jbyyBXmr#ntW@X~O+FoKEk zLW3}cJ6=Qp53W1#4}(5E{0@StKXB>~FIH$i^*!K!idHc_Vu}RS8y2Y12Cfboz(lh_ z$5?7QY#@V83lCwpnKw+LU?BV1LE`pmPJ4&v-LABQYlLO6JrXY>j<!zOt;3t4G;bOadEiu_M`KyZE?_wi`L=*_cP6WE<}r~CB^6H6 zqTpMLBth>XF7xuNl&6jrhpj&IKYdF})~KF?&jQB!8kPecjB>)&~;mY_lk;EuuiWj&LlJtTS`cZptv zaG%GRXH2#|k0K7J=Jdn2xn!}^?`LLzNV6Xv(hvSRrC)E>p`6e=#c4*_@^?+k{O|p; z<<}QGbnYyjD?Xbf?Fvi6v5G2tIahCG@|Lq%F_*mk@!C1qSL<@-{#;)35`9VeyRILe zDpx^7jCUoRc+M*{oT!ErkApQe5E}0hNc<%jP}}jX%dc+eXhn6GYy=!6>>%2IUB>fM z(%>Sv$|j@32ghLyIV(|vrIoGm&3x$Vy)Hi#c{4!VDAD<3E*P%}6Mug|mJ(xVgb*Tm zjM3i0!tD`e?WJ&tb%m(su6kDF8lA9DRNrnh@OW^`Foo*MvcVnvnHeEWiV-${@eyPvc}8c`iv%iClofQ>yyF+A2HR>nCqciu++WRC2j$5*-b{Dx8kw@M zVMJI1WEnG;+WLnfk@OXRb^7D&96J6kgFb z;mnH_)4Rx?UqRs$d{-LK4d6zCdvV|j;Gk8Bj6#8{>nsw!x7kFCjPtRMV3gd#7KK^v zNKYVo_K-S#7YanxR_QY61l+k)j~I@G3Ik<%5+_vJ2u>%l%`%1`xT;$bmSTams~2j> zQ_$NF4rvCCg-`c?J%exzuDXW5*~g8vgp7E3+C>7kjV{Ncuz!A5c^U4@wZ#qaiznP{ z@Ix6L>Ne)P?fE%_X6Mf}_;wpv{;|UJulTInuhsz!p&3u+i{MwF*tWENgUbQQ6r%Ek zS||KD5+Ju~(oS`ubzaA1Y=7m`ZjnEEc8p2cl@vLZ(dXKK8ZUMVilaZHqVBH3A4E9t z!A)86=aTqiIO|cGNU^Y5r>7Lv^(t>6_vj z)NKOZLKJ+!TfnXnl@`Dr!6LiC;1YgP(=Z3eZIU(db&aT%Of3n%U36XuG)L#16t{#> zi_)&E9;0-Bf}s|pp15o>meZaAU9}D2q^e12g8;>>=~V$)UM<04AH=CL8XTR_YhDH| z1H5R}(6F-II^tili#rfcPh9#Xv#qdJm(L*@ND1r zUxQX5) zuhnf;JNBVkTglfm-?pbC)jx1F_^{pN=Qp8sK=(yMSO>f3hQKu4$p=N3mF0t>PP+0z z=uaPiIK0r~jP<44Q;5L$E(WsX15m%XCSSmu%BCnR*O!q&Q|UIO=sRyK0+9d^{1+gY zjck9d_$|EwEeXZGkMpM4sIZ4Xl|$m_3XaQvx=XZNYvEYZUum zoV*CyO=zt=4Z1tME`bHCzU#;C>{&5jm03G^S1Z`&68goKCNSP4^vYMN2hPj$*JN)Bo z_lWn8SSW*=qpRAXW=rib{a}m)s@w_1wUn&a1F)bU{(F#SK=!?7gt@nn-jdgUv}h1G zXI*lVU({6;cM~lF9T1*x8!_3&>y@%hc`2@&cq$Md63f$H?wfdhSs#J_FpGeJ?mUgb zf6sRb7Dg(bNv!y7cfVT#=pAK^;XHBVB;c_987aXw6Gtx9DtIq?h$eF918`@##h&k{ z$ctCo9Q)xB*VV*9H|0XWniV;Jqf6QpG#QSXL&5K|Yi!)j1gxB8f6xa#YjZ$DO^&X3 zL^9GCV&kcJB6_=|iHX?ggoz2-w)3j|luJ9YBuiALxH zks5JZ&BL>Mb^`k0>{=8Ul=ZsvT#zjR-lJ!!aQu0M`+!RbAL$zot-17nJ@Q!R9M(=_ zUC3KIi*+t=y*mQs45!3*rnkq}|V zTx<=8jZ?7~w-fT{!bCm0%#W?UwDw{Yx zdu=)`IjpziYRG1}PuJj(u>iPP#YjyRE6eUHx~ebcx=E8pSx zvMUcglm{O-k1TE#uJdoVlJc~Dn}5w2w}tk+Of$_>?jpqqE};M`s9&M+la zL+genWqL#6@l|=lC6FllW@tn+s=+vQtfLMd3mYuEpG4E-G_ja&_ACe<{rd+Ga({Q!t|wsP#0$U)&U&S9>&Lv z!EPH9Gaj34PRwP0Zn8n4ergD)S08rYL;da#WUY_7bugnjve^1hBDWA04WyUhMHOT_ zVTQT!mV)k88ptYbYQ4>5?-uP=^VlQeJ!7&%9C$wdfDD2yBBTp(t~d1ggvbndG3?^v zc1sw4E<4(4rj;0KazXgt-ati7U(eWd*U>8_#A{2t~u-f4RGOA9gDv0`BVB}{r45T?+K zO%*1EdIF|@*`H00k$h_#47Samb9k}b8`tctX2`HHX26UdXa?-q(I%jO zbZ_K%cT-W4b<4UU)VipG#0OuQ%(}}UbDe*j>Zw5M;r>! zU~0iIs*5y>1cB>*;ghZ?Zg@S`T-yEytZR9&Rb&Jisha0u1@f<%ENrgUuM0dE3)LbK zZW%QqMlYvkbeLt;ECXmpunCD+X06Z>%dHt3X4y5%0GafX> z!8hbi(^}{&vT6G-^rnFS3?_<3SU02cFgDH@FxJu;V*%JZW4L&$XNDGK`;4)G=DRUu hq#ZPd0+|g)fGZ5x5qgNp^onq|v0)sJ{}0m6RlE~QlJx)p diff --git a/reference/agent/index.html b/reference/agent/index.html index 8c372908..f3144cbe 100644 --- a/reference/agent/index.html +++ b/reference/agent/index.html @@ -1148,6 +1148,27 @@ get_job_parameters() + + +
  • + + get_job_script_file() + + +
    • + +
  • + + process_supporting_files() + + +
    • + +
  • + + retrieve_submission_file() + +
  • @@ -1169,6 +1190,13 @@ unpack_error_from_slurm_response() +
    • + +
  • + + write_submission_file() + +
    • @@ -1861,6 +1889,27 @@ get_job_parameters() + + +
  • + + get_job_script_file() + + +
    • + +
  • + + process_supporting_files() + + +
    • + +
  • + + retrieve_submission_file() + +
  • @@ -1882,6 +1931,13 @@ unpack_error_from_slurm_response() +
    • + +
  • + + write_submission_file() + +
    • @@ -3372,6 +3428,93 @@
    + get_job_script_file + + + + async + + +
    +
    get_job_script_file(
+    pending_job_submission: PendingJobSubmission,
+    submit_dir: Path,
+) -> str
+
    + +
    + +

    Get the job script file from the backend.

    +

    Write the job script file to the submit_dir if WRITE_SUBMISSION_FILES is set to True.

    + +
    + + + + +
    + + + + +
    + process_supporting_files + + + + async + + +
    +
    process_supporting_files(
+    pending_job_submission: PendingJobSubmission,
+    submit_dir: Path,
+)
+
    + +
    + +

    Process the submission support files.

    +

    Write the support files to the submit_dir if WRITE_SUBMISSION_FILES is set to True. +Reject the submission if there are support files with WRITE_SUBMISSION_FILES set to False.

    + +
    + +
    + + +
    + + + + +
    + retrieve_submission_file + + + + async + + +
    +
    retrieve_submission_file(file: JobScriptFile) -> str
+
    + +
    + +

    Get a submission file from the backend and return the decoded file content.

    + +
    + +
    + + +
    + + + +
    submit_job_script @@ -3449,6 +3592,34 @@
    +
    + + + + +
    + write_submission_file + + + + async + + +
    +
    write_submission_file(
+    file_content: str, filename: str, submit_dir: Path
+)
+
    + +
    + +

    Write a decoded file content to the submit_dir.

    + +
    + +
    + +
    diff --git a/search/search_index.json b/search/search_index.json index f9986a0c..50cad235 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":"

    An Omnivector initiative

    "},{"location":"#jobbergate-documentation","title":"Jobbergate Documentation","text":"

    The following documentation provides a comprehensive overview of the Jobbergate system, detailing its purpose, installation process, and operational guidelines.

    Jobbergate serves as an advanced job templating and submission system, designed to seamlessly integrate with Slurm. This integration facilitates the efficient re-use and remote submission of job scripts to a Slurm cluster.

    At the heart of Jobbergate is its API, which acts as the pivotal control center for the entire system. This API interacts with an agent positioned alongside a Slurm cluster. This agent is responsible for establishing communication between both the Jobbergate API and the Slurm RESTful API. Furthermore, Jobbergate offers a Command Line Interface (CLI) to ensure users have an intuitive means of interacting with the system.

    Given that the API is cloud-based, users are granted the capability to modify jobs, dispatch them to affiliated clusters, and oversee their progress from any device with internet connectivity.

    Additionally, Jobbergate introduces a Python SDK named \"Jobbergate Core\". This SDK is equipped with tools tailored for automation and can be effortlessly integrated into any Python-based project.

    "},{"location":"authors/","title":"Authors","text":"

    Jobbergate is written and maintained by Omnivector, LLC. It is an open source project developed in collaboration with Scania, AB.

    "},{"location":"authors/#attribution","title":"Attribution","text":"

    This project began as a rewrite of the original Jobbergate project authored by Jimmy Hedman. The original provided the inspiration for an interactive tool used to gather template variable values to render jinja2 templates into Slurm job scripts.

    Building upon Jimmy's great idea, Jobbergate has grown into an entire system for managing and submitting reusable Slurm jobs, but the core idea of reusing templates combined with user input for the template values has remained the core of the project throughout its evolution.

    "},{"location":"authors/#jobbergate-development-team","title":"Jobbergate Development Team","text":"

    The Jobbergate project's main contributors are as follows:

    • Felipe Schuch \ud83d\udce7
    • James Beedy \ud83d\udce7
    • Lucas Carvalho
    • Matheus Tosta \ud83d\udce7
    • Tucker Beck \ud83d\udce7
    "},{"location":"authors/#get-in-touch","title":"Get in touch","text":"

    Contact Omnivector by email

    "},{"location":"tutorial/","title":"Tutorial","text":"

    Welcome to this step-by-step tutorial that introduces the basic functionalities of Jobbergate! to seamlessly upload a Job Script and submit it to a Slurm cluster using the Jobbergate CLI.

    In this walk-through, you will learn how to upload a Job Script and submit it to a Slurm cluster using the Jobbergate CLI. To accomplish this, we will guide you through the following steps:

    • Initiating a session by logging into the Jobbergate system
    • Uploading a basic Job Script to Jobbergate
    • Submitting the Job Script to the cluster
    • Reviewing the results and monitoring the status of your submitted job
    • Cleaning up by deleting the Job Script
    • Logging out of the Jobbergate system
    "},{"location":"tutorial/#getting-started","title":"Getting Started","text":"

    Before diving into the tutorial, there are some initial setup steps that are needed to ensure that your computer is prepared to run the tutorial locally. Make sure you have administrative access to your machine, as it's required for the setup process.

    "},{"location":"tutorial/#install-docker-compose","title":"Install docker-compose","text":"

    For this tutorial, we will be using an instance of Jobbergate that is deployed locally along-side a local Slurm cluster. We will set all this up using docker-compose. If you do not have it already, follow this guide to install docker-compose before you continue the tutorial.

    "},{"location":"tutorial/#update-the-hostfile","title":"Update the hostfile","text":"

    Next, you\u2019ll need to add the following line to your computer\u2019s hostfile:

    127.0.0.1 keycloak.local\n
    "},{"location":"tutorial/#for-linux-and-osx-users","title":"For Linux and OSX users","text":"
    • The hostfile is located at /etc/hosts.
    • Open a terminal.

    • Use this command to open the file in a text editor

      sudo nano /etc/hosts\n

      Note

      You may, of course, substitute nano by your editor of choice

    • Add the above line at the end of the file.

    • Save and close the file.
    "},{"location":"tutorial/#for-windows-users","title":"For Windows users","text":"
    • The hostfile can be found at c:\\windows\\system32\\drivers\\etc\\hosts.
    • Open Notepad as an administrator.
    • Open the hostfile in Notepad.
    • Append the above line to the file.
    • Save and exit.
    "},{"location":"tutorial/#clone-jobbergate-with-git","title":"Clone Jobbergate with git","text":"

    To run the Jobbergate and Slurm locally, you will first need a copy of the Jobbergate source code. The easiest way to get it is to use Git to download the source code repository from GitHub onto your machine.

    Git is a version control system that lets you manage and keep track of your source code history. If you haven't installed it yet, download and install Git using the instructions available here.

    With Git installed, you can now clone the Jobbergate source code from its GitHub repository. Cloning allows you to have a local copy (or clone) of the source code on your machine.

    Run the following command in your terminal:

    git clone git@github.com:omnivector-solutions/jobbergate.git\n

    Now you have a full copy of the Jobbergate source code including the Docker Compose configuration to stand up a local Slurm Cluster and the example Job Script we will be using for this tutorial.

    Next, switch to the directory in the source code that contains the Docker Compose configuration:

    cd jobbergate/jobbergate-composed\n
    "},{"location":"tutorial/#start-the-jobbergate-services","title":"Start the Jobbergate Services","text":"

    With the Jobbergate source code in place, it's time to initiate the Jobbergate Services and the local Slurm cluster using Docker Compose. Follow the steps outlined below to get things up and running.

    "},{"location":"tutorial/#start-up-the-services","title":"Start up the services","text":"

    Run the following command to build and start the services. The --build flag ensures that Docker Compose build the images before attempting to start the services. The --detach flag runs the services in the background so that you can run other commands in the terminal.

    docker-compose up --build --detach\n

    This operation might take a few minutes as it involves building the images and starting up all the associated services.

    "},{"location":"tutorial/#verify-the-status-of-the-services","title":"Verify the status of the services","text":"

    To confirm that all the services are running smoothly, execute the following command. It will list the status of all the services initiated by Docker Compose:

    docker-compose ps\n

    If the services are up and running as expected, you should see output similar to the following, indicating that all the services are in a healthy state

    NAME                                        COMMAND                  SERVICE               STATUS              PORTS\nc1                                          \"/usr/local/bin/slur\u2026\"   c1                    running             6818/tcp\nc2                                          \"/usr/local/bin/slur\u2026\"   c2                    running             6818/tcp\njobbergate-composed-cluster-agent-1         \"/agent/entrypoint.sh\"   cluster-agent         running\njobbergate-composed-db-1                    \"docker-entrypoint.s\u2026\"   db                    running             0.0.0.0:5432->5432/tcp\njobbergate-composed-jobbergate-api-1        \"/bin/sh -c /app/dev\u2026\"   jobbergate-api        running (healthy)   0.0.0.0:8000->80/tcp\njobbergate-composed-jobbergate-cli-1        \"python3\"                jobbergate-cli        exited (0)\njobbergate-composed-keycloak.local-1        \"/opt/keycloak/bin/k\u2026\"   keycloak.local        running             0.0.0.0:8080->8080/tcp, 8443/tcp\njobbergate-composed-minio-1                 \"/usr/bin/docker-ent\u2026\"   minio                 running             0.0.0.0:9000-9001->9000-9001/tcp\njobbergate-composed-minio-create-bucket-1   \"/create-bucket.sh\"      minio-create-bucket   exited (1)\nmysql                                       \"docker-entrypoint.s\u2026\"   mysql                 running             3306/tcp, 33060/tcp\nslurmctld                                   \"/usr/local/bin/slur\u2026\"   slurmctld             running             6817/tcp\nslurmdbd                                    \"/usr/local/bin/slur\u2026\"   slurmdbd              running             6819/tcp\nslurmrestd                                  \"/usr/local/bin/slur\u2026\"   slurmrestd            running             0.0.0.0:6820->6820/tcp\n

    The STATUS for each service should be \"running\" except for the minio-create-bucket and jobbergate-cli services that should be \"exited\".

    "},{"location":"tutorial/#confirm-jobbergate-cli-availability","title":"Confirm Jobbergate CLI availability","text":"

    Since this tutorial relies on running commands in the Jobbergate CLI, it's essential to verify that the CLI is available and working as expected at this juncture.

    First, initiate a connection to the jobbergate-cli container by executing the following command. This gives you direct access to the CLI.

    docker-compose run jobbergate-cli bash\n

    Upon successful connection, your command prompt should change to reflect that you're inside the container. It will look something like this:

    root@e226a9a401d1:/app#\n

    This confirms that you're now operating within the jobbergate-cli container environment.

    Next, we need to make sure that the Jobbergate CLI is available and accepting commands. Test this by listing the available commands in Jobbergate CLI with the --help option:

    jobbergate --help\n

    The command above will yield a detailed description of the CLI's usage and the variety of sub-commands it provides:

     Usage: jobbergate [OPTIONS] COMMAND [ARGS]...\n\n Welcome to the Jobbergate CLI!\n More information can be shown for each command listed below by running it with the --help option.\n\n\u256d\u2500 Options \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 --verbose               --no-verbose                                     Enable verbose logging to the terminal      \u2502\n\u2502                                                                          [default: no-verbose]                       \u2502\n\u2502 --full                  --no-full                                        Print all fields from CRUD commands         \u2502\n\u2502                                                                          [default: no-full]                          \u2502\n\u2502 --raw                   --no-raw                                         Print output from CRUD commands as raw json \u2502\n\u2502                                                                          [default: no-raw]                           \u2502\n\u2502 --version               --no-version                                     Print the version of jobbergate-cli and     \u2502\n\u2502                                                                          exit                                        \u2502\n\u2502                                                                          [default: no-version]                       \u2502\n\u2502 --install-completion                    [bash|zsh|fish|powershell|pwsh]  Install completion for the specified shell. \u2502\n\u2502                                                                          [default: None]                             \u2502\n\u2502 --show-completion                       [bash|zsh|fish|powershell|pwsh]  Show completion for the specified shell, to \u2502\n\u2502                                                                          copy it or customize the installation.      \u2502\n\u2502                                                                          [default: None]                             \u2502\n\u2502 --help                                                                   Show this message and exit.                 \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n\u256d\u2500 Commands \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 applications        Commands to interact with applications                                                           \u2502\n\u2502 job-scripts         Commands to interact with job scripts                                                            \u2502\n\u2502 job-submissions     Commands to interact with job submissions                                                        \u2502\n\u2502 login               Log in to the jobbergate-cli by storing the supplied token argument in the cache.                \u2502\n\u2502 logout              Logs out of the jobbergate-cli. Clears the saved user credentials.                               \u2502\n\u2502 show-token          Show the token for the logged in user.                                                           \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n
    "},{"location":"tutorial/#log-in-to-jobbergate","title":"Log in to Jobbergate","text":"

    To begin working with Jobbergate data, you must first sign into the system. For the purpose of this tutorial, there's just one user available. We'll solely focus on this user in this guide, but should you wish to add more users, you can do so by accessing the Keycloak server (details provided in the Appendix).

    To log in using the Jobbergate CLI, execute the following command:

    jobbergate login\n

    The CLI will provide a URL for you to log into your account:

    \u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Waiting for login \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502                                                                                                                   \u2502\n\u2502   To complete login, please open the following link in a browser:                                                 \u2502\n\u2502                                                                                                                   \u2502\n\u2502     http://keycloak.local:8080/realms/jobbergate-local/device?user_code=CZAU-TZAH                                 \u2502\n\u2502                                                                                                                   \u2502\n\u2502   Waiting up to 5.0 minutes for you to complete the process...                                                    \u2502\n\u2502                                                                                                                   \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n\nWaiting for web login... \u2501\u257a\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501   3% 0:04:50\n

    Open the URL shown in a browser and log in as \"local-user\":

    • username: \"local-user\"
    • password: \"local\"

    When prompted, grant all the requested access privileges to the CLI. Once you have finished, the CLI will show that you have successfully logged in:

    \u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Logged in! \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502                                                                                                                   \u2502\n\u2502   User was logged in with email 'local-user@jobbergate.local'                                                     \u2502\n\u2502                                                                                                                   \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n

    You are now logged in through the CLI! Your auth token will be cached automatically for you, so you should not need to log in again for some time. However, be aware that your session does expire; you will have to log in again to get a new token. If this happens, the CLI will alert you that your token is invalid. When you receive this notification, you will need to log in anew.

    "},{"location":"tutorial/#upload-a-job-script-to-jobbergate","title":"Upload a Job Script to Jobbergate","text":"

    Job Scripts are integral to Jobbergate, serving as the foundation for running simulations on our cluster. To initiate a simulation, your first task is to upload the Job Script to the Jobbergate API.

    Within each Job Script, an entrypoint file is designated. This is the specific script that Slurm executes to commence the simulation on the cluster.

    "},{"location":"tutorial/#get-the-example-script","title":"Get the example script","text":"

    To keep this tutorial focused on using Jobbergate and not any of the complexities of simulations or operating a cluster, we will use a very basic example job script. We will need a copy of this script where the jobbergate-cli can access it. Since it's a small script, we can just copy/paste it into the container where we are accessing the jobbergate-cli.

    In the terminal where you were typing jobbergate commands, enter this command:

    cat > simple-job-script.py\n

    Paste the contents of the job script and then press ctrl-d on your keyboard. This will create a saved copy of the job script that's ready to submit with the jobbergate-cli. To ensure that the command sequence captured the intended script contents, execute the following command to review the job script:

    cat simple-job-script.python3\n

    The script should appear exactly as you see it on the link above.

    "},{"location":"tutorial/#create-the-job-script-from-the-example","title":"Create the Job Script from the example","text":"

    Now it's time to create a Job Script entry within the Jobbergate system. We'll use the create subcommand associated with the job-scripts command. To view all the options that come with this sub-command, you can use the --help option:

    jobbergate job-scripts create --help\n

    Now, let's create the Job Script. In your terminal, type:

    jobbergate job-scripts create --name=tutorial --job-script-path=simple-job-script.py\n

    You should see output like this indicating that the Job Script was successfully created:

                     Created Job Script\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key            \u2503 Value                            \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 id             \u2502 1                                \u2502\n\u2502 application_id \u2502 None                             \u2502\n\u2502 name           \u2502 tutorial                         \u2502\n\u2502 description    \u2502                                  \u2502\n\u2502 owner_email    \u2502 local-user@jobbergate.local-mail \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n

    Great, your Job Script is now prepared and ready for submission to the cluster!

    Note

    Keep track of the id value produced by your command. The tutorial text assumes that it is \"1\", but it may be different if you have done the tutorial before or had to restart!

    To confirm that the Job Script has been uploaded correctly, you can review the file content using the show-files subcommand:

    jobbergate job-scripts show-files --id=1\n

    The file should appear exactly as it does on the link above.

    "},{"location":"tutorial/#submit-a-job-script-to-the-cluster","title":"Submit a Job Script to the cluster","text":"

    With the Job Script ready, the next step is to submit it to the Slurm cluster. In this tutorial, a cluster named local-slurm is already attached and available for use. We will specify this cluster name when submitting the Job Script to ensure it is executed on the appropriate cluster.

    "},{"location":"tutorial/#create-the-job-submission","title":"Create the Job Submission","text":"

    We will use the create subcommand of the job-submissions command to submit the job to the cluster. To see all the options available for this command, we can use the --help option again:

    jobbergate job-submissions create --help\n

    For the tutorial, we need to issue the following command:

    jobbergate job-submissions create --name=tutorial --job-script-id=1 --cluster-name=local-slurm\n

    The command should produce output that looks like this:

                       Created Job Submission\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key                        \u2503 Value                       \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 id                         \u2502 1                           \u2502\n\u2502 job_script_id              \u2502 1                           \u2502\n\u2502 client_id                  \u2502 local-slurm                 \u2502\n\u2502 slurm_job_id               \u2502 None                        \u2502\n\u2502 execution_directory        \u2502 None                        \u2502\n\u2502 job_submission_name        \u2502 tutorial                    \u2502\n\u2502 job_submission_description \u2502 None                        \u2502\n\u2502 job_submission_owner_email \u2502 local-user@jobbergate.local \u2502\n\u2502 status                     \u2502 CREATED                     \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n

    Info

    The Job Submission was successfully created! However, it has not submitted to the cluster yet, and thus slurm_job_id is still None. This will happen when the Jobbergate Agent that is running remotely in the cluster pulls all \"CREATED\" Job Submissions down from the API and submits them to Slurm one by one.

    Note

    Again, be careful to use the correct id produced by this command for the remainder of the tutorial!

    "},{"location":"tutorial/#check-the-status-of-the-submitted-job","title":"Check the status of the submitted job","text":"

    We can look up the status of a Job Submission using the following command:

    jobbergate job-submissions get-one --id=1\n

    This command should produce output that looks like:

                           Job Submission\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key                        \u2503 Value                       \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 id                         \u2502 1                           \u2502\n\u2502 job_script_id              \u2502 1                           \u2502\n\u2502 client_id                  \u2502 local-slurm                 \u2502\n\u2502 slurm_job_id               \u2502 1                           \u2502\n\u2502 execution_directory        \u2502 None                        \u2502\n\u2502 job_submission_name        \u2502 tutorial                    \u2502\n\u2502 job_submission_description \u2502 None                        \u2502\n\u2502 job_submission_owner_email \u2502 local-user@jobbergate.local \u2502\n\u2502 status                     \u2502 SUBMITTED                   \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n

    If the status reported by your command is CREATED, don't worry! The Jobbergate Agent just hasn't retrieved and submitted the job script yet. Wait a few more seconds and try again. You should now see the status change to SUBMITTED.

    When the Job Submission status shifts to SUBMITTED, it indicates that the Jobbergate Agent has retrieved the Job Script and submitted it to the local-slurm cluster. This status will persist until the completion of the Job Script's execution. The Jobbergate Agent continuously monitors the job's progress within slurm, and, upon its completion, will update the Job Submission status to COMPLETE.

    "},{"location":"tutorial/#check-the-results-of-the-job","title":"Check the results of the job","text":"

    In this tutorial, we have locally mounted a \"fake\" NFS folder to contain the output from the job running in slurm. When the job finishes running, it will produce an output file in this folder. First we need to verify that the file was produced by listing the contents of the nfs directory:

    ls /nfs\n

    If the job completed, you should see a file in the /nfs directory named simple-output.txt. Check the contents of the file with a simple cat command:

    cat /nfs/simple-output.txt\n

    It should look look like:

    Simple output from c1\n

    It's possible that the output says it came from c2 if slurm ran the job on the c2 compute node instead of c1.

    "},{"location":"tutorial/#delete-the-resources","title":"Delete the resources","text":"

    Sometimes it is useful to remove resources that have been created in Jobbergate.

    For instance, start by deleting the Job Submission:

    $ jobbergate job-submissions delete --id=1\n\n\u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Job submission delete succeeded \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502                                                                                                                   \u2502\n\u2502   The job submission was successfully deleted.                                                                    \u2502\n\u2502                                                                                                                   \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n

    Then delete the Job Script:

    $ jobbergate job-scripts delete --id=1\n\n\u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Job script delete succeeded \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502                                                                                                                   \u2502\n\u2502   The job script was successfully deleted.                                                                        \u2502\n\u2502                                                                                                                   \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n
    "},{"location":"tutorial/#log-out-of-the-jobbergate-system","title":"Log out of the Jobbergate system","text":"

    You have completed the tutorial. Try logging out of Jobbergate now:

    $ jobbergate logout\n\n\u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Logged out \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502                                                                                                                   \u2502\n\u2502   User was logged out.                                                                                            \u2502\n\u2502                                                                                                                   \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n

    This will clear any cached tokens, and any subsequent Jobbergate commands will require you to log in again.

    "},{"location":"tutorial/#appendix","title":"Appendix","text":""},{"location":"tutorial/#keycloak-ui","title":"Keycloak UI","text":"

    You can connect to the Keycloak UI to create additional realms, clients, and users. However, the use of Keycloak is a rather large topic that goes outside the scope of this Tutorial.

    To get started, you can connect to the Keycloak UI through a browser if the server is running as a part of the docker-compose cluster using this local URL. To log in as administrator use these credentials:

    • username: admin
    • password: admin
    "},{"location":"developer_guide/ci/","title":"Continuous Integration","text":"

    Jobbergate employs GitHub actions for its continuous integration processes. Detailed descriptions of these actions are provided on this page.

    "},{"location":"developer_guide/ci/#automated-quality-assurance","title":"Automated Quality Assurance","text":"

    Jobbergate's git repository incorporates a GitHub Action, specified in test_on_push.yaml, which is designed to execute our quality assurance tools across all Jobbergate sub-projects simultaneously. The action is activated anytime a new commit is pushed to the main branch or whenever a pull request is submitted.

    The suite of quality assurance tools encompasses unit tests, code coverage, linters, code formatters, and static type checkers. Comprehensive documentation about each tool is available in the Quality Assurance Tools section.

    "},{"location":"developer_guide/ci/#automated-publication-to-pypi","title":"Automated Publication to PyPI","text":"

    The major components of Jobbergate are published on PyPI, the Python Package Index. They are available at:

    • jobbergate-api
    • jobbergate-cli
    • jobbergate-agent
    • jobbergate-core

    These packages are automatically published to PyPI by three linked GitHub Actions that are detailed below.

    "},{"location":"developer_guide/ci/#prepare-for-release","title":"Prepare for release","text":"

    The first action involved in publication is the prepare_release.yaml) action. It is triggered manually on github through a \"workflow dispatch event\" whenever new features or fixes need to be published.

    The action takes two arguments that must be supplied by the user. They are:

    • Use workflow from: The branch from which the release will be created. The default is main, and it's highly recommended that releases are cut from this branch in order to keep a linear commit history between releases and pre-releases.
    • Release Type: This will describe the release type that will be created. Because Jobbergate uses semantic versioning, it's important to carefully select the correct type of release. For mor information on release types, please see the Poetry documentation to learn more.

    Once activated, this action:

    • Uses Poetry to bump the version number of all the Jobbergate sub-packages according to the release type selected.
    • Checks if the new version number is synchronized between the sub-packages, and fails if they are not.
    • Creates a new dated entry for the new release on each of the sub-packages' changelog files from the contents of the \"Unreleased\" section.
    • Creates a new branch named prepare-release/<version>.
    • Opens a draft pull request titled Release <version>.

    In this way, all the changes above can be reviewed before the release is published, and all quality assurance tests are executed for the pull request.

    The remaining steps of the workflow are chained automatically once the PR is accepted and merged into main.

    "},{"location":"developer_guide/ci/#create-a-new-tag","title":"Create a new tag","text":"

    The next action in the sequence is the tag_on_merged_pull_request.yaml action. Once the automatically created release PR is merged into the main branch, this action is triggered. It creates and pushes a new git tag to GitHub. The tag is based on the new version number for the release.

    "},{"location":"developer_guide/ci/#publish-on-tag","title":"Publish on Tag","text":"

    The final action is publish_on_tag.yaml This action is triggered when a new version tag is pushed to the repository. It first double checks if the tag matches the version number of each Jobbergate component, and then it builds and publishes the packages on PyPI.

    "},{"location":"developer_guide/dev_tools/","title":"API Dev Tools","text":"

    The Jobbergate API sub-project is equipped with a few tools designed to assist with some everyday development tasks. These can help streamline the process of setting up and interacting with the API.

    The dev-tools are shipped as a CLI program that can be invoked via Poetry within the project. All of the commands will operate within the virtual environment set up by Poetry.

    "},{"location":"developer_guide/dev_tools/#invoking-dev-tools","title":"Invoking dev-tools","text":"

    To invoke the dev tools, you must execute the commands from the home directory for the jobbergate-api. To see some information about the dev-tools, execute:

    poetry run dev-tools --help\n

    This will provide some help output that shows what options and sub-commands are available:

    Usage: dev-tools [OPTIONS] COMMAND [ARGS]...\n\nOptions:\n  --install-completion [bash|zsh|fish|powershell|pwsh]\n                                  Install completion for the specified shell.\n  --show-completion [bash|zsh|fish|powershell|pwsh]\n                                  Show completion for the specified shell, to\n                                  copy it or customize the installation.\n  --help                          Show this message and exit.\n\nCommands:\n  db\n  dev-server  Start a development server locally.\n  show-env    Print out the current environment settings.\n

    The --help option is available for all of the subcommands provided in dev-tools.

    "},{"location":"developer_guide/dev_tools/#the-db-subcommand","title":"The db subcommand","text":"

    There are a few convenience methods in the dev-tools for interacting with Jobbergate API's PostgreSQL database. These tools are found in the db subcommand. To see more info about this sub-command, run:

    poetry run dev-tools db --help\n
    "},{"location":"developer_guide/dev_tools/#the-login-subcommand","title":"The login subcommand","text":"

    This command allows you to log in to the database that your Jobbergate API is configured to connect with. It allows you to login to databases, regardless of whether they are locally hosted via Docker or situated on a remote PostgreSQL server. this ensures seamless access to any database that the Jobbergate API is configured to connect with.

    To log in to the database, execute this command:

    poetry run dev-tools db login\n

    The command will show some debug output including the URL of the database to which it is connecting and will then show a REPL connection to the database:

    2022-09-07 15:52:02.089 | DEBUG    | dev_tools.db:login:26 - Logging into database: postgresql://compose-db-user:compose-db-pswd@localhost:5432/compose-db-name\nServer: PostgreSQL 14.1 (Debian 14.1-1.pgdg110+1)\nVersion: 3.4.1\nHome: http://pgcli.com\ncompose-db-name>\n
    "},{"location":"developer_guide/dev_tools/#the-migrate-subcommand","title":"The migrate subcommand","text":"

    This command uses alembic to generate a migration script to bring the current database (described by the environment) up to date with the SQLAlchemy models specified in the Jobbergate API source code.

    To invoke the migration script generation, execute:

    poetry run dev-tools db migrate --message=\"An example migration\"\n

    Some logging info will be produced, including the location of the new migration script:

    2022-09-07 15:58:09.725 | DEBUG    | dev_tools.db:migrate:79 - Creating migration with message: An example migration\nINFO  [alembic.runtime.migration] Context impl PostgresqlImpl.\nINFO  [alembic.runtime.migration] Will assume transactional DDL.\nINFO  [alembic.ddl.postgresql] Detected sequence named 'applications_id_seq' as owned by integer column 'applications(id)', assuming SERIAL and omitting\nINFO  [alembic.ddl.postgresql] Detected sequence named 'job_scripts_id_seq' as owned by integer column 'job_scripts(id)', assuming SERIAL and omitting\nINFO  [alembic.ddl.postgresql] Detected sequence named 'job_submissions_id_seq' as owned by integer column 'job_submissions(id)', assuming SERIAL and omitting\n  Generating /home/dusktreader/git-repos/omnivector/jobbergate/jobbergate-api/alembic/versions/20220907_155809--c275de463a90_an_example_migration.py ...  done\n  Running post write hook \"black\" ...\nreformatted /home/dusktreader/git-repos/omnivector/jobbergate/jobbergate-api/alembic/versions/20220907_155809--c275de463a90_an_example_migration.py\n\nAll done! \u2728 \ud83c\udf70 \u2728\n1 file reformatted.\n  done\n  Running post write hook \"isort\" ...\nFixing /home/dusktreader/git-repos/omnivector/jobbergate/jobbergate-api/alembic/versions/20220907_155809--c275de463a90_an_example_migration.py\n  done\n

    The generated migration should always be reviewed before it is committed to the repository.

    It is also possible to produce a blank migration if you need to execute some raw SQL or write an Alembic script by hand. Just pass the --blank parameter on the command line:

    poetry run dev-tools db migrate --blank --message=\"A blank migration\"\n
    "},{"location":"developer_guide/dev_tools/#the-upgrade-subcommand","title":"The upgrade subcommand","text":"

    This subcommand is used to apply a database migration to the database that the Jobbergate API is configured to connect with.

    By default, it will apply all the migrations that have not yet been applied to the database.

    To apply the migrations, execute the command:

    poetry run dev-tools db upgrade\n

    It will produce some logging output that shows what migrations were applied:

    2022-09-07 16:05:46.315 | DEBUG    | dev_tools.db:upgrade:89 - Upgrading database...\nINFO  [alembic.runtime.migration] Context impl PostgresqlImpl.\nINFO  [alembic.runtime.migration] Will assume transactional DDL.\nINFO  [alembic.runtime.migration] Running upgrade d22da0741b7f -> c275de463a90, An example migration\n

    If you wish to only upgrade the database to a specific migration, you can pass that migration's id to the --target param.

    "},{"location":"developer_guide/dev_tools/#the-show-env-subcommand","title":"The show-env subcommand","text":"

    This command will show how the Jobbergate API is configured through its environment settings. To see the environment, execute this command:

    poetry run dev-tools show-env\n

    The output that the command produces will look something like:

    Jobbergate settings:\n  DEPLOY_ENV: LOCAL\n  LOG_LEVEL: DEBUG\n  DATABASE_HOST: localhost\n  DATABASE_USER: compose-db-user\n  DATABASE_PSWD: compose-db-pswd\n  DATABASE_NAME: compose-db-name\n  DATABASE_PORT: 5432\n  TEST_DATABASE_HOST: localhost\n  TEST_DATABASE_USER: test-user\n  TEST_DATABASE_PSWD: test-pswd\n  TEST_DATABASE_NAME: test-db\n  TEST_DATABASE_PORT: 5433\n  S3_BUCKET_NAME: jobbergate-k8s-staging\n  S3_ENDPOINT_URL: None\n  ARMASEC_DOMAIN: localhost:9080/realms/master/protocol/openid-connect\n  ARMASEC_USE_HTTPS: True\n  ARMASEC_AUDIENCE: https://local.omnivector.solutions\n  ARMASEC_DEBUG: True\n  ARMASEC_ADMIN_DOMAIN: None\n  ARMASEC_ADMIN_AUDIENCE: None\n  ARMASEC_ADMIN_MATCH_KEY: None\n  ARMASEC_ADMIN_MATCH_VALUE: None\n  IDENTITY_CLAIMS_KEY: https://omnivector.solutions\n  SENTRY_DSN: None\n  SENTRY_SAMPLE_RATE: 1.0\n  MAX_UPLOAD_FILE_SIZE: 104857600\n  SENDGRID_FROM_EMAIL: None\n  SENDGRID_API_KEY: None\n

    The command can also produce the output as JSON if needed by passing the --json flag:

    poetry run dev-tools show-env --json\n

    The JSON output will look something like:

    {\"DEPLOY_ENV\": \"LOCAL\", \"LOG_LEVEL\": \"DEBUG\", \"DATABASE_HOST\": \"localhost\", \"DATABASE_USER\": \"compose-db-user\", \"DATABASE_PSWD\": \"compose-db-pswd\", \"DATABASE_NAME\": \"compose-db-name\", \"DATABASE_PORT\": 5432, \"TEST_DATABASE_HOST\": \"localhost\", \"TEST_DATABASE_USER\": \"test-user\", \"TEST_DATABASE_PSWD\": \"test-pswd\", \"TEST_DATABASE_NAME\": \"test-db\", \"TEST_DATABASE_PORT\": 5433, \"S3_BUCKET_NAME\": \"jobbergate-k8s-staging\", \"S3_ENDPOINT_URL\": null, \"ARMASEC_DOMAIN\": \"localhost:9080/realms/master/protocol/openid-connect\", \"ARMASEC_USE_HTTPS\": true, \"ARMASEC_AUDIENCE\": \"https://local.omnivector.solutions\", \"ARMASEC_DEBUG\": true, \"ARMASEC_ADMIN_DOMAIN\": null, \"ARMASEC_ADMIN_AUDIENCE\": null, \"ARMASEC_ADMIN_MATCH_KEY\": null, \"ARMASEC_ADMIN_MATCH_VALUE\": null, \"IDENTITY_CLAIMS_KEY\": \"https://omnivector.solutions\", \"SENTRY_DSN\": null, \"SENTRY_SAMPLE_RATE\": 1.0, \"MAX_UPLOAD_FILE_SIZE\": 104857600, \"SENDGRID_FROM_EMAIL\": null, \"SENDGRID_API_KEY\": null}\n
    "},{"location":"developer_guide/dev_tools/#the-dev-server-subcommand","title":"The dev-server subcommand","text":"

    This command starts up a local development server for the Jobbergate API. It will be created using the configuration set up in the environment settings. This command is especially useful if you want to run the API locally but connect to remote services such as a database and s3 hosted on AWS.

    To start the server, run:

    poetry run dev-tools dev-server\n

    The command will produce some logging output that looks like this:

    2022-09-07 16:15:05.830 | INFO     | dev_tools.dev_server:dev_server:50 - Waiting for the database\n2022-09-07 16:15:05.830 | DEBUG    | dev_tools.dev_server:_wait_for_db:23 - database url is: postgresql://compose-db-user:compose-db-pswd@localhost:5432/compose-db-name\n2022-09-07 16:15:05.830 | DEBUG    | dev_tools.dev_server:_wait_for_db:26 - Checking health of database at postgresql://compose-db-user:compose-db-pswd@localhost:5432/compose-db-name: Attempt #0\nINFO:     Will watch for changes in these directories: ['/home/dusktreader/git-repos/omnivector/jobbergate/jobbergate-api']\nINFO:     Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)\nINFO:     Started reloader process [27314] using statreload\n2022-09-07 16:15:06.555 | INFO     | jobbergate_api.main:<module>:39 - Skipping Sentry\nINFO:     Started server process [27319]\nINFO:     Waiting for application startup.\n2022-09-07 16:15:06.587 | INFO     | jobbergate_api.main:init_logger:71 - Logging configured \ud83d\udcdd Level: DEBUG\n2022-09-07 16:15:06.587 | DEBUG    | jobbergate_api.main:init_database:79 - Initializing database\nINFO:     Application startup complete.\n

    There are additional options that can control some of the details of the settings of the dev server. These can be examined with the --help flag:

    poetry run dev-tools dev-server --help\n

    The dev server options will be printed like:

    Usage: dev-tools dev-server [OPTIONS]\n\n  Start a development server locally.\n\nOptions:\n  --db-wait-count INTEGER   How many times to attempt a check  [default: 3]\n  --db-wait-interval FLOAT  Seconds to wait between checks  [default: 5.0]\n  --port INTEGER            The port where the server should listen  [default:\n                            5000]\n  --log-level TEXT          The level to log uvicorn output  [default: DEBUG]\n  --help                    Show this message and exit.\n

    The --db-wait-* flags are used to make the dev server wait for the dev database to become available. These are mostly useful in the context of docker-compose.

    It should also be noted that a development uvicorn server will automatically reload the app if the source files of the app change. This is very helpful for debugging behavior in the app without having to manually stop and start the app after every source code modification.

    "},{"location":"developer_guide/integration_testing/","title":"Integration Testing","text":"

    While conducting integration testing for Jobbergate, it's critical to examine the entire cycle of the platform, ranging from the creation of an Application to remote Job Submission via the Jobbergate Agent.

    To test most of the platforms functionality, the docker-compose setup located in the Jobbergate Composed sub-project is sufficient. Begin by referring to the guide in that sub-project's README. Pay close attention to the execution of Jobbergate CLI commands as they play a significant role in integration testing. For testing you can use the pre-configured user credentials:

    • Username: local-user
    • Password: local

    Integration testing should cover the following work-flows:

    • Logging in through the CLI
    • Creating an Application
    • Querying a single Application
    • Updating an Application
    • Rendering a Job Script from an Application
    • Updating a Job Script
    • Submitting a Job Script
    • Verifying a Job Submission
    • Deleting the Job Submission, Job Script, and Application
    • Logging out through the CLI
    "},{"location":"developer_guide/integration_testing/#setup","title":"Setup","text":"

    To begin, you will need two separate terminals open. Change directory to the jobbergate-composed sub-project of the top-level jobbergate folder.

    First, you need to start up the Jobbergate platform with docker-compose. In one of your terminals, run the following command:

    docker-compose up --build\n

    Once all the services are started, jump into the prepared jobbergate-cli container to execute CLI commands. To do so, execute this command in the other terminal you have prepared:

    docker-compose run jobbergate-cli bash\n

    Now you may start executing commands with the Jobbergate CLI.

    To assist with some of the commands below, create a NAME environment variable that will help to identify resources that you create during the process. You should set the value based on the current date so that the associated resources are easy to identify. Run the following command to set it:

    export NAME=\"test--$(whoami)--$(date -I)\"\n

    You have now created a test name like test--tbeck--2023-10-13.

    "},{"location":"developer_guide/integration_testing/#logging-in-through-the-cli","title":"Logging in through the CLI","text":"

    The first work-flow you will test covers the auth mechanics of both the CLI and the API.

    Run the following command in the Jobbergate CLI:

    jobbergate login\n

    Next, open the link that is printed out and log in as local-user (password \"local\"). If asked, grant all of the permissions.

    Verify that the CLI reports that the user has been successfully logged in.

    At this point, verify that the token that has been retrieved for the user is correct.

    Run the following command in the CLI:

    jobbergate show-token --decode\n

    This command will pretty print the payload of the token. Verify that it contains:

    • \"view\" and \"edit\" permissions for job-templates, job-scripts, and job-submissions
    • email equalling \"local-user@jobberate.local\"
    • aud includes \"https://local.omnivector.solutions\"
    • azp equals \"jobbergate-cli\"
    "},{"location":"developer_guide/integration_testing/#creating-an-application","title":"Creating an Application","text":"

    Next, test the command to create an Application through the CLI, and verify that the resource is created in the database. Also, verify that the files are successfully uploaded to the file store.

    For integration testing, use the built-in simple application. example. This example application has 3 simple template variables, and, when submitted, the rendered Job Script simply prints the values of those variables.

    Run the following command in the Jobbergate CLI:

    jobbergate applications create --name=$NAME --identifier=$NAME --application-path=/example\n

    Verify that output shows that a single application was inserted and that the files were uploaded:

                        Created Application\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key                  \u2503 Value                            \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 id                   \u2502 1                                \u2502\n\u2502 name                 \u2502 test--root--2023-10-13           \u2502\n\u2502 owner_email          \u2502 local-user@jobbergate.local-mail \u2502\n\u2502 is_archived          \u2502 False                            \u2502\n\u2502 description          \u2502                                  \u2502\n\u2502 identifier           \u2502 test--root--2023-10-13           \u2502\n\u2502 application_uploaded \u2502 True                             \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n
    "},{"location":"developer_guide/integration_testing/#querying-a-single-application","title":"Querying a single Application","text":"

    Next, verify that we can look up a single Application by both its id and its identifier. Also include the --full argument to the base jobbergate command so that the output will show all the fields in the database including the source file, the config, and the timestamps.

    First, fetch the Application by id using the following command in the CLI:

    jobbergate --full applications get-one --id=1\n

    The output should look something like this:

    \u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key            \u2503 Value                                                                                                                                 \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 id             \u2502 1                                                                                                                                     \u2502\n\u2502 name           \u2502 test--root--2023-10-13                                                                                                                \u2502\n\u2502 owner_email    \u2502 local-user@jobbergate.local-mail                                                                                                      \u2502\n\u2502 created_at     \u2502 2023-10-13T19:44:21.935886                                                                                                            \u2502\n\u2502 updated_at     \u2502 2023-10-13T19:44:21.958325                                                                                                            \u2502\n\u2502 identifier     \u2502 test--root--2023-10-13                                                                                                                \u2502\n\u2502 description    \u2502                                                                                                                                       \u2502\n\u2502 template_vars  \u2502 {'bar': 'BAR', 'baz': 'BAZ', 'foo': 'FOO', 'workdir': '/nfs'}                                                                         \u2502\n\u2502 template_files \u2502 [{'parent_id': 5, 'filename': 'dummy-script.py.j2', 'file_type': 'ENTRYPOINT', 'created_at': '2023-10-13T19:44:22.020542',            \u2502\n\u2502                \u2502 'updated_at': '2023-10-13T19:44:22.020556'}]                                                                                          \u2502\n\u2502 workflow_files \u2502 [{'parent_id': 5, 'filename': 'jobbergate.py', 'runtime_config': {'template_files': None, 'job_script_name': None,                    \u2502\n\u2502                \u2502 'default_template': 'dummy-script.py.j2', 'output_directory': '.', 'supporting_files': None, 'supporting_files_output_name': None},   \u2502\n\u2502                \u2502 'created_at': '2023-10-13T19:44:22.110739', 'updated_at': '2023-10-13T19:44:22.110750'}]                                              \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n

    Verify that the id, name, identifier, and timestamps match the Application that was created.

    Next, fetch the same application by identifier and verify that it is the same Application:

    jobbergate --full applications get-one --identifier=test--tbeck--2023-10-13\n
    "},{"location":"developer_guide/integration_testing/#updating-an-application","title":"Updating an Application","text":"

    Next, verify that you can update the application through the CLI.

    Run this command to verify that we can change the name:

    jobbergate applications update --id=1 --application-desc=\"Here is a test description\"\n

    Verify that you can see the new description in the application when you fetch it via the get-one subcommand. Also, check the output with the --full parameter to make sure that the updated_at field is different and later than the created_at field.

    "},{"location":"developer_guide/integration_testing/#rendering-a-job-script-from-an-application","title":"Rendering a Job Script from an Application","text":"

    Now that an application has been uploaded uploaded, use it to render a new Job Script.

    There are a few different options to test here to check for correct behavior:

    • Basic, interactive render
    • Render in \"fast mode\" with a --param-file
    • Render with additional SBATCH params
    "},{"location":"developer_guide/integration_testing/#basic-interactive-render","title":"Basic, interactive render","text":"

    First, render an Application to a Job Script by executing the interactive code that gathers the values for template variables from the user.

    To start the rendering process, execute:

    jobbergate job-scripts render --name=$NAME --application-id=1\n

    Verify that you are shown 3 prompts to supply values for the template variables. Fill these in with any values you like. Notice that the third question has a default response supplied already. Accept this value or replace it with your preferred value:

       [?] gimme the foo!: FOO\n   [?] gimme the bar!: BAR\n   [?] gimme the foo!: BAZ\n   [?] gimme the workdir!: /nfs\n

    When prompted if you would like to submit the job, decline with \"n\".

    After completing the questions, verify that the CLI reports that the new Job Script was created using the supplied values:

                     Created Job Script\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key            \u2503 Value                            \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 id             \u2502 1                                \u2502\n\u2502 application_id \u2502 1                                \u2502\n\u2502 name           \u2502 test--root--2023-10-13           \u2502\n\u2502 description    \u2502                                  \u2502\n\u2502 owner_email    \u2502 local-user@jobbergate.local-mail \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n

    Now we need to verify that the Job Script was rendered with the correct values. Run the following command:

    jobbergate job-scripts show-files --id=1\n

    The output should show the Job Script with the provided template variable values rendered as expected:

    \u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 dummy-script.py \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502                                                                                                                                                        \u2502\n\u2502   #!/bin/python3                                                                                                                                       \u2502\n\u2502                                                                                                                                                        \u2502\n\u2502   #SBATCH -J dummy_job                                                                                                                                 \u2502\n\u2502   #SBATCH -t 60                                                                                                                                        \u2502\n\u2502                                                                                                                                                        \u2502\n\u2502   print(\"Executing dummy job script\")                                                                                                                  \u2502\n\u2502   with open(\"/nfs/dummy-output.txt\", mode=\"w\") as dump_file:                                                                                           \u2502\n\u2502       print(\"I am a very, very dumb job script\", file=dump_file)                                                                                       \u2502\n\u2502       print(\"foo=FOO\", file=dump_file)                                                                                                                 \u2502\n\u2502       print(\"bar=BAR\", file=dump_file)                                                                                                                 \u2502\n\u2502       print(\"baz=BAZ\", file=dump_file)                                                                                                                 \u2502\n\u2502                                                                                                                                                        \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 This is the main job script file \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n
    "},{"location":"developer_guide/integration_testing/#render-in-fast-mode-with-a-param-file","title":"Render in \"fast mode\" with a --param-file","text":"

    Next, verify that a Job Script can be rendered while skipping the interactive question answering segment by pre-supplying the application with the values to use for rendering. Since only the third question has a default, supply at least the other two questions with a param using the --param-file parameter.

    First, create a file to hold the params (hit ctrl-d to finish and write the file):

    cat > params.json\n{\n  \"foo\": \"FOO\",\n  \"bar\": \"BAR\"\n}\n

    Now, render the Application using this file. Include the --no-submit flag because the Job Script shouldn't be submitted immediately. Verify only the rendering process for the new Job Script:

    jobbergate job-scripts render --name=$NAME --application-id=1 --fast --param-file=params.json --no-submit\n

    The output from the command will show you the default values that were used that you did not specify in the params.json file:

    Default values used\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key     \u2503 Value \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 baz     \u2502 zab   \u2502\n\u2502 workdir \u2502 /nfs  \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n\n\n                 Created Job Script\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key            \u2503 Value                            \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 id             \u2502 2                                \u2502\n\u2502 application_id \u2502 1                                \u2502\n\u2502 name           \u2502 test--root--2023-10-13           \u2502\n\u2502 description    \u2502                                  \u2502\n\u2502 owner_email    \u2502 local-user@jobbergate.local-mail \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n

    Now check the rendered file again using the show-files sub-command.

    "},{"location":"developer_guide/integration_testing/#render-with-additional-sbatch-params","title":"Render with additional SBATCH params","text":"

    Finally, test that additional SBATCH params can be inserted at render time. The code will insert these additional parameters into the rendered Job Script files.

    To supply extra SBATCH params, they are provided on the command line using the --sbatch-params option. Use this command to test it out:

    jobbergate job-scripts render --name=$NAME --application-id=1 --fast --param-file=params.json --no-submit --sbatch-params=\"--cluster=fake\" --sbatch-params=\"--partition=dummy\"\n

    The output should look like:

    Default values used\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key     \u2503 Value \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 baz     \u2502 zab   \u2502\n\u2502 workdir \u2502 /nfs  \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n\n\n                 Created Job Script\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key            \u2503 Value                            \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 id             \u2502 3                                \u2502\n\u2502 application_id \u2502 1                                \u2502\n\u2502 name           \u2502 test--root--2023-10-13           \u2502\n\u2502 description    \u2502                                  \u2502\n\u2502 owner_email    \u2502 local-user@jobbergate.local-mail \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n

    Now, review the rendered Job Script file using the --show-files command. You should see that the additional two SBATCH parameters are included:

    #SBATCH --cluster=fake                                                                                                                               \u2502\n#SBATCH --partition=dummy\n
    "},{"location":"developer_guide/integration_testing/#updating-a-job-script","title":"Updating a Job Script","text":"

    Next, verify that an existing Job Script can be updated.

    Run this command to verify that you can change the description:

    jobbergate job-scripts update --id=1 --description=\"Here is a test description\"\n

    Verify that you can see the new description in the Job Script when you fetch it via the get-one subcommand. Also, check the output with the --full parameter to make sure that the updated_at field is different and later than the created_at field.

    "},{"location":"developer_guide/integration_testing/#submitting-a-job-script","title":"Submitting a Job Script","text":"

    Next, test the process of submitting a Job Script to a slurm cluster for execution. Note that the docker-compose.yaml used for testing sets up a volume-mounted directory named /nfs. The /nfs directory in the container is mounted from the slurm-fake-nfs directory in the jobbergate-composed subproject. You can look in this directory after the job completes execution to check the results.

    You will need to verify that jobs are being submitted correctly vai the following steps:

    • Submit the job through the CLI
    • Verify that the agent submitted the job
    • Verify that the agent updates the Job Submission status
    • Verify the output from the job
    "},{"location":"developer_guide/integration_testing/#submit-the-job-through-the-cli","title":"Submit the job through the CLI","text":"

    Submit the Job Script using the CLI by running the following command:

    jobbergate job-submissions create --name=$NAME --job-script-id=1 --cluster-name=local-slurm --execution-directory=/nfs\n

    Verify that the output shows that the Job Submission has been created for the target Job Script

                      Created Job Submission\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key                 \u2503 Value                            \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 id                  \u2502 1                                \u2502\n\u2502 job_script_id       \u2502 1                                \u2502\n\u2502 cluster_name        \u2502 local-slurm                      \u2502\n\u2502 slurm_job_id        \u2502 None                             \u2502\n\u2502 execution_directory \u2502 /nfs                             \u2502\n\u2502 name                \u2502 test--root--2023-10-13           \u2502\n\u2502 description         \u2502                                  \u2502\n\u2502 owner_email         \u2502 local-user@jobbergate.local-mail \u2502\n\u2502 status              \u2502 CREATED                          \u2502\n\u2502 report_message      \u2502 None                             \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n
    "},{"location":"developer_guide/integration_testing/#verify-that-the-agent-submitted-the-job","title":"Verify that the agent submitted the job","text":"

    To verify that the agent submitted the job correctly, review the log output from the agent.

    The agent performs the following process to complete a Job Submission with Slurm

    • Fetch a pending job from the API
    • Submit the job to slurm
    • Mark the job as submitted and add the slurm job id to it

    You can access the log data by running the following command in a terminal that has changed directory to the jobbergate-composed folder:

    docker-compose logs jobbergate-agent\n

    It may be useful to pipe the output to a text viewer like less.

    If the agent has successfully submitted the job, you should see some log lines that look like this (ellipses indicate omitted content):

    ...Retrieved 1 pending job submission...\n...Submitting pending job submission 1\n...\n...Submitting pending job submission 1 to slurm...\n...\n...Received slurm job id 1 for job submission 1\n...Marking job job_submission_id=22 as SUBMITTED (slurm_job_id=1)\n

    If you find those log lines, then the Agent has successfully submitted the job to slurm.

    "},{"location":"developer_guide/integration_testing/#verify-that-the-job-was-completed","title":"Verify that the job was completed","text":"

    To verify that the job completed successfully, review the log output from the agent.

    The agent performs the following process to complete Job Submissions

    • Fetch the submitted job from the API
    • Check the status of the job in slurm using the the slurm_job_id
    • If the job is completed, mark the Job Submission as COMPLETED

    You should look for log lines that look like this (ellipses indicate omitted content):

    ...Retrieved 1 active job submissions...\n...Fetching status of job_submission 1 from slurm\n...Fetching slurm job status for slurm job 1\n...\n...Status for slurm job 1 is job_id=1 job_state='COMPLETED'...\n...Updating job_submission with status=COMPLETED\n
    "},{"location":"developer_guide/integration_testing/#verify-the-output-from-the-job","title":"Verify the output from the job","text":"

    In the terminal where you are running Jobbergate CLI commands, you can check the /nfs directory to see the results. You should see three output files in the directory:

    • test--root--2023-10-13.out
    • test--root--2023-10-13.err
    • dummy-output.txt

    First, check the standard output from the script:

    cat /nfs/test--root--2023-10-13.out\n

    You should see a single line that says:

    Executing dummy job script\n

    The standard error from the script should be empty.

    The final file, dummy-output.txt, should contain the following content:

    I am a very, very dumb job script\nfoo=FOO\nbar=BAR\nbaz=BAZ\n
    "},{"location":"developer_guide/integration_testing/#conclusion","title":"Conclusion","text":"

    The process described in this document covers integration tests across the entire Jobbergate platform. These integration tests should be performed before new versions of Jobbergate are published and before Jobbergate is deployed to a new environment.

    "},{"location":"developer_guide/keycloak_setup/","title":"Setting up Keycloak for Jobbergate","text":"

    Jobbergate's security is provided by the Armasec package which should be compatible with any OIDC provider. However, the recommended provider is Keycloak.

    In this guide, we outline the steps to integrate an existing Keycloak instance (version 19.0.2 as used in this example) with Jobbergate to ensure a smooth user experience and enhanced security features.

    Although this tutorial focuses on integrating Keycloak with a locally deployed instance of Jobbergate, such as one housed in a Docker container via the jobbergate-composed sub-project, the procedures can be easily adapted to suit deployments on single-node Keycloak clusters or other complex configurations.

    "},{"location":"developer_guide/keycloak_setup/#create-a-new-realm","title":"Create a new Realm","text":"

    You have the option to utilize an existing realm for Jobbergate, but for a streamlined process, it's typically more advantageous to create a new realm specifically for your Jobbergate deployment.

    Once you're logged into the Keycloak interface, navigate and click the Add realm button, found beneath the Select realm dropdown menu on the left-hand side.

    For those using a local Jobbergate deployment, you should assign the Name as \"jobbergate-local\".

    You'll also need to specify a Frontend URL. Avoid using \"localhost\" because a valid domain is required for the redirection to function correctly. A suitable alternative is the .local special domain; this domain is ideal as it isn't subject to reservation on any DNS. For instance, your full Frontend URL would be http://keycloak.local:8080.

    The remaining realm settings can be left at their default configurations.

    "},{"location":"developer_guide/keycloak_setup/#setup-hostfile","title":"Setup Hostfile","text":"

    For the Keycloak admin UI to work correctly in a local deployment, it's essential to include the keycloak.local domain in your system\u2019s hostfile.

    For users on Linux or OSX, you can find this file at /etc/hosts. Windows users can locate it at c:\\windows\\system32\\drivers\\etc\\hosts.

    Editing this file requires administrative or sudo privileges.

    Upon accessing the file, append the following line and save your changes:

    127.0.0.1   keycloak.local\n

    This step ensures that the Frontend URL resolves correctly, facilitating seamless navigation and operation.

    "},{"location":"developer_guide/keycloak_setup/#create-a-new-client-for-the-cli","title":"Create a new Client for the CLI","text":"

    To facilitate login and JWT authentication for the Jobbergate CLI, it's essential to allocate a dedicated client.

    Begin by navigating to the Clients section, found on the left sidebar, and then proceed to click on the Create button located on the right.

    When adjusting the Client Protocol settings, select the openid-connect option. For the Client ID setting, which choosing an easy to identify name like \"jobbergate-cli\" is best even though this field can be any unique string.

    To ensure the CLI can utilize this client for login purposes, it's vital to activate the OAuth 2.0 Device Authorization Grant option.

    If you're working with a local deployment, simply input \"*\" in the Valid Redirect URIs section.

    "},{"location":"developer_guide/keycloak_setup/#add-roles","title":"Add Roles","text":"

    Next, we need to add the needed roles for Jobbergate endpoints. These represent the fine-grained permissions that are checked for each request to make sure that the user has permission to fulfill the request.

    Click the Roles tab at the top, and then click on Add Role on the right.

    Add the following roles:

    Name Description jobbergate:job-templates:edit Allow to view job templates jobbergate:job-templates:view Allow to view job templates jobbergate:job-scripts:edit Allow to edit job scripts jobbergate:job-scripts:view Allow to view job scripts jobbergate:job-submissions:edit Allow to edit job submissions jobbergate:job-submissions:view Allow to view job submissions"},{"location":"developer_guide/keycloak_setup/#add-mappers","title":"Add Mappers","text":"

    Jobbergate requires two claims that are not available by default. We will add them to the JWTs with Mappers.

    Click the Mappers tab at the top, and then click the Create button to add a new Mapper.

    "},{"location":"developer_guide/keycloak_setup/#audience","title":"Audience","text":"

    First, we need to add an \"audience\" mapper. Select \"audience\" for the Name field. Next, select \"Audience\" for the Mapper Type. The Included Custom Audience value may be whatever you like. The local deploy, by default, uses https://apis.omnivector.solutions. Make sure to enable the Add to ID token setting.

    "},{"location":"developer_guide/keycloak_setup/#permissions","title":"Permissions","text":"

    The Armasec package expects to find \"permissions\" in a claim at the root of the JWT payload. To facilitate this, we need to add a mapper that will copy the permissions to the correct place in the JWT. We will call the new mapper our \"permissions\" mapper.

    Enter \"Permissions\" under the Name field. Next, select \"User Client Role\" as the Mapper Type. Select \"jobbergatel-cli\" for the Client ID. The Token Claim Name must have the value \"permissions\". The Claim JSON Type field must be \"String\".

    "},{"location":"developer_guide/keycloak_setup/#create-a-new-client-for-the-agent","title":"Create a new Client for the Agent","text":"

    The Jobbergate Agent also requires its own client.

    Again, click the Clients section on the left navigation bar, and then click the Create button on the right.

    For the Client Protocol setting, choose the openid-connect protocol. The Client ID setting will be used to match jobs to the cluster they should be submitted to. So use the cluster name for this setting. For a local deployment, the Client ID should be \"local-slurm\".

    On the Settings tab, set Access Type to confidential and enter \"*\" for the Valid Redirect URIs. Scroll down and click on the Save button.

    "},{"location":"developer_guide/keycloak_setup/#add-roles_1","title":"Add Roles","text":"

    Click on the Roles tab, and click the Add Role button. Add all the following roles as above:

    Name Description jobbergate:job-templates:edit Allow to view Jobbergate applications jobbergate:job-templates:view Allow to view applications jobbergate:job-scripts:edit Allow to edit job scripts jobbergate:job-scripts:view Allow to view job scripts jobbergate:job-submissions:edit Allow to edit job submissions jobbergate:job-submissions:view Allow to view job submissions"},{"location":"developer_guide/keycloak_setup/#add-mappers_1","title":"Add Mappers","text":"

    Like the CLI client, the Agent's client also requires the \"Audience\" and \"Permissions\" mappers.

    Click the Mappers tab at the top, and then click the Create button to add a new Mapper.

    "},{"location":"developer_guide/keycloak_setup/#audience_1","title":"Audience","text":"

    First, we need to add an \"audience\" mapper. Select \"audience\" for the Name field. Next, select \"Audience\" for the Mapper Type. The Included Custom Audience value may be whatever you like. The local deploy, by default, uses \"https://apis.omnivector.solutions\". Make sure to enable the Add to ID token setting.

    "},{"location":"developer_guide/keycloak_setup/#permissions_1","title":"Permissions","text":"

    Next, add a \"permissions\" mapper. The Armasec package expects to find a \"permissions\" claims under a claim at the root of the JWT payload. Enter \"Permissions\" under the Name field. Next, select \"User Client Role\" as the Mapper Type. Select \"jobbergatel-cli\" for the Client ID. The Token Claim Name must have the value \"permissions\". The Claim JSON Type field must be \"String\".

    "},{"location":"developer_guide/keycloak_setup/#add-service-account-roles","title":"Add Service Account Roles","text":"

    To add the correct roles to the tokens issued for the Agent's client, we need to add some \"Service Account Roles\".

    Click the Service Account Roles tab. Then, from the Client Roles drop-down, select the local-slurm client. Select all of the Jobbergate roles created above and then click the Add selected button.

    "},{"location":"developer_guide/keycloak_setup/#create-users","title":"Create User(s)","text":"

    You will need to create some users that can use Jobbergate. These users will be able to sign-in through the Jobbergate CLI. Each user must have a unique email address. Other than that, no special settings are needed.

    To add a user, click Users on the left nav bar. Next, click the Add user button on the right.

    Use the following settings, and then click the Save button.

    | Username | local-user | | Email | local-user@jobbergate.local | | First Name | Local | | Last Name | User |

    After you have created the user, edit it by clicking on it in the list. You may need to click on the View all users button to see it.

    Click the Credentials tab at the top. Enter \"local\" for the Password and Password Confirmation field. Turn the Temporary setting to OFF, and click Reset Password. Click the Set password verification button.

    Next, click the Role Mappings tab at the top. Select the jobbergate-local entry in the Client Roles drop-down. Select all of the roles for jobbergate added above and click Add selected to add them to the user.

    "},{"location":"developer_guide/keycloak_setup/#conclusion","title":"Conclusion","text":"

    Your Keycloak instance is now prepared for use by Jobbergate! For additional information on configuring Keycloak and Armasec, consult documentation at:

    • https://www.keycloak.org/documentation
    • https://omnivector-solutions.github.io/armasec/
    "},{"location":"developer_guide/qa_tools/","title":"Quality Assurance Tools","text":"

    Jobbergate utilizes quality control tools across its three primary components (API, CLI, and Agent). The tools are invoked in the same way in each of the sub-projects, and may be invoked en masse from the root Jobbergate directory.

    "},{"location":"developer_guide/qa_tools/#running-unit-tests","title":"Running Unit Tests","text":"

    The main sub-projects each make use of pytest to apply unit testing. The unit tests for each are contained in a subdirectory named tests/.

    To invoke all of the unit tests for a sub-project, simply issue the following command:

    make test\n

    Once you enter the command above, the unit tests suite will start running. For the API, this process takes a few minutes. The others only take a few seconds. The status of the tests will be logged to the console as well as a coverage report for the unit tests:

    ================================================================== test session starts ===================================================================\nplatform linux -- Python 3.8.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0\nUsing --random-order-bucket=module\nUsing --random-order-seed=650699\n\nrootdir: /home/dusktreader/git-repos/omnivector/jobbergate/jobbergate-api, configfile: pyproject.toml, testpaths: jobbergate_api/tests\nplugins: asyncio-0.12.0, random-order-1.0.4, respx-0.17.1, env-0.6.2, armasec-0.11.0, freezegun-0.4.2, cov-2.12.1, anyio-3.5.0\ncollecting ... 2022-09-07 16:31:37.548 | INFO     | jobbergate_api.main:<module>:39 - Skipping Sentry\ncollected 158 items\n\ntests/apps/job_scripts/test_routers.py ........................                                                                     [ 15%]\ntests/apps/applications/test_schemas.py ....                                                                                        [ 17%]\ntests/test_file_validation.py ...........                                                                                           [ 24%]\ntests/test_email_notification.py .......                                                                                            [ 29%]\ntests/apps/applications/test_application_files.py .........                                                                         [ 34%]\ntests/apps/job_submissions/test_routers.py .................................                                                        [ 55%]\ntests/apps/job_scripts/test_job_script_files.py .........                                                                           [ 61%]\ntests/apps/test_main.py .                                                                                                           [ 62%]\ntests/test_meta_mapper.py ...                                                                                                       [ 63%]\ntests/test_s3_manager.py ...                                                                                                        [ 65%]\ntests/test_config.py ................                                                                                               [ 75%]\ntests/test_pagination.py ........                                                                                                   [ 81%]\ntests/test_storage.py ..                                                                                                            [ 82%]\ntests/test_security.py ...                                                                                                          [ 84%]\ntests/apps/applications/test_routers.py .........................                                                                   [100%]\n\n==================================================================== warnings summary ====================================================================\ntests/conftest.py:53\n  /home/dusktreader/git-repos/omnivector/jobbergate/jobbergate-api/tests/conftest.py:53: PytestUnknownMarkWarning: Unknown pytest.mark.enforce_empty_database - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/mark.html\n    @pytest.mark.enforce_empty_database()\n\ntests/apps/job_scripts/test_routers.py: 37 warnings\ntests/apps/job_submissions/test_routers.py: 40 warnings\ntests/test_pagination.py: 10 warnings\ntests/apps/applications/test_routers.py: 42 warnings\n  /home/dusktreader/.cache/pypoetry/virtualenvs/jobbergate-api-zc2JKxO9-py3.8/lib/python3.8/site-packages/databases/backends/postgres.py:114: DeprecationWarning: The `Row.keys()` method is deprecated to mimic SQLAlchemy behaviour, use `Row._mapping.keys()` instead.\n    warnings.warn(\n\n-- Docs: https://docs.pytest.org/en/stable/warnings.html\n\n---------- coverage: platform linux, python 3.8.12-final-0 -----------\nName                                                               Stmts   Miss  Cover   Missing\n------------------------------------------------------------------------------------------------\njobbergate_api/__init__.py                                             0      0   100%\njobbergate_api/apps/__init__.py                                        0      0   100%\njobbergate_api/apps/applications/__init__.py                           0      0   100%\njobbergate_api/apps/applications/application_files.py                 77      0   100%\njobbergate_api/apps/applications/models.py                             7      0   100%\njobbergate_api/apps/applications/routers.py                          136     14    90%   64-66, 129-131, 136-137, 210-212, 331-332, 341\njobbergate_api/apps/applications/schemas.py                           66      0   100%\njobbergate_api/apps/job_scripts/__init__.py                            0      0   100%\njobbergate_api/apps/job_scripts/job_script_files.py                  121      4    97%   67, 80-81, 270\njobbergate_api/apps/job_scripts/models.py                              7      0   100%\njobbergate_api/apps/job_scripts/routers.py                           132     11    92%   98-100, 108-109, 235-237, 267-269, 301\njobbergate_api/apps/job_scripts/schemas.py                            38      0   100%\njobbergate_api/apps/job_submissions/__init__.py                        0      0   100%\njobbergate_api/apps/job_submissions/constants.py                      11      0   100%\njobbergate_api/apps/job_submissions/models.py                          8      0   100%\njobbergate_api/apps/job_submissions/routers.py                       186     12    94%   101-103, 260-262, 382, 395-400, 406, 449\njobbergate_api/apps/job_submissions/schemas.py                        51      0   100%\njobbergate_api/apps/permissions.py                                     8      0   100%\njobbergate_api/config.py                                              58      1    98%   102\njobbergate_api/email_notification.py                                  28      0   100%\njobbergate_api/file_validation.py                                    102      6    94%   36-56, 111, 175\njobbergate_api/main.py                                                47      4    91%   31-37, 94\njobbergate_api/meta_mapper.py                                         24      1    96%   104\njobbergate_api/metadata.py                                             2      0   100%\njobbergate_api/pagination.py                                          31      0   100%\njobbergate_api/s3_manager.py                                          14      0   100%\njobbergate_api/security.py                                            22      0   100%\njobbergate_api/storage.py                                             52      1    98%   128\ntests/__init__.py                                       0      0   100%\ntests/apps/__init__.py                                  0      0   100%\ntests/apps/applications/__init__.py                     0      0   100%\ntests/apps/applications/test_application_files.py     104      0   100%\ntests/apps/applications/test_routers.py               368      0   100%\ntests/apps/applications/test_schemas.py                14      0   100%\ntests/apps/conftest.py                                 41      0   100%\ntests/apps/job_scripts/__init__.py                      0      0   100%\ntests/apps/job_scripts/conftest.py                     10      2    80%   32, 49\ntests/apps/job_scripts/test_job_script_files.py       102      0   100%\ntests/apps/job_scripts/test_routers.py                373      3    99%   48-64, 72\ntests/apps/job_submissions/__init__.py                  0      0   100%\ntests/apps/job_submissions/test_routers.py            483      0   100%\ntests/apps/test_main.py                                 7      0   100%\ntests/conftest.py                                     114      1    99%   127\ntests/test_config.py                                   33      0   100%\ntests/test_email_notification.py                       44      0   100%\ntests/test_file_validation.py                          17      0   100%\ntests/test_meta_mapper.py                              27      0   100%\ntests/test_pagination.py                               55      0   100%\ntests/test_s3_manager.py                               17      0   100%\ntests/test_security.py                                 39      0   100%\ntests/test_storage.py                                   7      0   100%\n------------------------------------------------------------------------------------------------\nTOTAL                                                               3083     60    98%\n\nRequired test coverage of 95.0% reached. Total coverage: 98.05%\n=========================================================== 158 passed, 130 warnings in 52.46s ===========================================================\n

    Note

    The API unit tests require that a test database is already running. You can start one by using the dev-tools provided in the API sub-project.

    "},{"location":"developer_guide/qa_tools/#running-linters","title":"Running Linters","text":"

    The main sub-projects each use a group of linting tools to make sure that the code follows code quality standards. These linters will report any lines or segments of the code that do not meet the project's standards.

    To invoke all of the linters for a sub-project, issue the following command:

    make lint\n

    If any issues are reported, fix the reported error and try running it again. The linters will only succeed if all of the issues are fixed.

    "},{"location":"developer_guide/qa_tools/#running-formatters","title":"Running Formatters","text":"

    For most of the linting issues, the code can be auto-corrected using the configured code formatters.

    Currently, the sub-projects use the following formatters:

    • black
    • isort

    To apply the formatters, use this command:

    make format\n

    The formatters will report any files that were changed in their reports.

    "},{"location":"developer_guide/qa_tools/#running-static-code-checkers","title":"Running Static Code Checkers","text":"

    The Jobbergate sub-projects include type-hints that must be checked using the mypy static code checker. It may invoked using make:

    make mypy\n

    If any issues are located, they will be reported. Each type issue must be fixed before the static type checker passes.

    "},{"location":"developer_guide/qa_tools/#running-all-quality-checks","title":"Running All Quality Checks","text":"

    Finally, all of the quality checks can be run using this command:

    make qa\n
    "},{"location":"developer_guide/template_workflows/","title":"Job Script Template Workflow Files","text":"

    The main workflow file is a python script that is used within an interactive framework that gathers the values for template variables that will be needed when Job Scripts are rendered from Applications.

    Throughout the documentation, this file is referred to as the \"Workflow Source File.\"

    The entire purpose of the Workflow Source File is to construct a workflow of questions organized in a series of that can be changed dynamically according to the answers provided by the user.

    "},{"location":"developer_guide/template_workflows/#the-jobbergateapplication-class","title":"The JobbergateApplication class","text":"

    Each Workflow Source File script must define exactly one class named JobbergateApplication.

    This class should be a regular python class that inherits from the JobbergateApplicationBase. This base class is imported from the application_base module.

    The JobbergateApplication implementation may be a simple or complex as needed by the user. However, it must define a mainflow() method which is the first of the workflow methods that the Application processes.

    "},{"location":"developer_guide/template_workflows/#the-workflow-methods","title":"The workflow methods","text":"

    The mainflow() method is essentially the entry point for the Workflow Source File. It must return a list of questions that should be asked to the user in order. These questions will be used to gather the template variable values.

    The mainflow() method must take a dictionary named data as a keyword argument. This kwarg should default to None, and it should be set to an empty dict if the default is not overridden.

    Each workflow can also specify the net workflow method to call after its questions have been asked and answered. In this way, the workflows can be organized in a dynamic series where the path is dictated by the user responses.

    The workflow methods specify the next flow in the sequence by setting an item keyed by \"nextworkflow\" in the data dictionary. The value of this item is the name of the next workflow method to call.

    Each workflow method can examine the results from previous workflows by referencing the data dict. All of the key/value pairs in the dictionary (besides \"nextworkflow\") represent answers to previous questions.

    "},{"location":"developer_guide/template_workflows/#the-questions","title":"The Questions","text":"

    The Workflow Source File is built around a question asking framework that defines different sorts of questions that can be asked of the user.

    The question types are defined by classes that derive from a base QuestionBase class. The question types include:

    • Text: gather a simple string response from the user
    • Integer: gather a simple int response from the user
    • List: prompt the user to select one item from a list
    • Directory: prompt the user for a directory path
    • File: prompt the user for a file path
    • Checkbox: prompt the user to select as many items from a list as they want
    • Confirm: prompt the user to offer a boolean response
    • BooleanList: prompt a series of boolean responses
    • Const: set the variable to the default value without even asking the user

    Note

    The BooleanList question has some very complex logic. The source code should be examined to understand what this does in detail.

    All of the implementation of the question classes (including the base class) can be found in the questions module of the Jobbergate source code.

    "},{"location":"developer_guide/template_workflows/#other-class-attributes","title":"Other class attributes","text":"

    Each Workflow Source File also has access to some attributes set up by the JobbergateApplicationBase.

    The jobbergate_config attribute will contain any of the properties that are set in the jobbergate_config section of the Application Config (jobbergate.yaml). These values can include anything set up by the user at application creation time.

    The application_config attribute contains all of the properties that are set in the application_config section of the Application config (jobbergate.yaml). This section may be empty. If it is, the application_config attribute will be an empty dictionary. This dictionary should only be populated by the template variables that the Workflow Source File seeks to collect from the user. The values for each item are the default values for that template variable.

    "},{"location":"elements/apps/","title":"Jobbergate Apps","text":"

    Jobbergate consists of three interconnected Python applications that operate harmoniously. These applications enable the creation and dispatch of Job Scripts to a Slurm cluster, eliminating the need for the Jobbergate user to engage directly with Slurm \u2013 a process that might be challenging or unfeasible.

    While the primary interface for user interaction with Jobbergate is the CLI, both the API and Core package can be employed to develop automation and craft tools leveraging Jobbergate's capabilities.

    The three apps in Jobbergate are:

    • Jobbergate API
    • Jobbergate CLI
    • Jobbergate Agent

    And the SDK that provides python integration is:

    • Jobbergate Core
    "},{"location":"elements/apps/agent/","title":"Jobbergate Agent Overview","text":"

    The Jobbergate Agent is a daemon application that is designed to be integrated into the slurm cluster.

    It predominantly fulfills two key roles:

    • Submitting newly created Job Submissions to the Slurm cluster
    • Monitoring and updating the status of Job Submissions as they undergo processing
    "},{"location":"elements/apps/agent/#submitting-jobs","title":"Submitting Jobs","text":"

    The Jobbergate Agent constantly monitors the Job Submissions resource for entries marked with a CREATED status. These are Job Submissions that the API has instantiated but are yet to be dispatched to Slurm.

    When submitting a job to Slurm, the Jobbergate Agent pulls the Job Script itself plus any supporting files associated with it down to the cluster. Once all the files have been downloaded, the Job Script is submitted to Slurm via it's RESTful API. The Job Submission saves the identifier for the Slurm Job so that it can be associated with the Job Script that was submitted. The Job Submission also tracks all of the supporting files and submission parameters that were submitted along with the Job Script.

    Upon job submission to Slurm, the Jobbergate Agent retrieves not only the Job Script but also any related supporting files, downloading them to the cluster. After ensuring all files are downloaded, the Job Script is dispatched to Slurm through its RESTful API. The Job Submission retains the unique identifier for the Slurm Job, ensuring it's linked to the submitted Job Script. Additionally, the Job Submission logs all the supporting files and submission parameters that were provided in tandem with the Job Script at submission time.

    "},{"location":"elements/apps/agent/#updating-job-status","title":"Updating Job Status","text":"

    Once submitted, the Jobbergate Agent updates the status of the Job Submission to SUBMITTED. If there is an error during the submission process, the Agent sets the Job Submission status to REJECTED.

    Upon completion of the job by the Slurm cluster, the Agent updates the status either to COMPLETE if successful, or FAILED if the job could not complete. This signifies the conclusion of tasks related to that particular Job Submission.

    "},{"location":"elements/apps/agent/#usage","title":"Usage","text":"

    The Jobbergate Agent operates in the background; it's designed to be initiated and left uninterrupted.

    For insights into its ongoing operations, the Agent offers detailed logging which can be analyzed.

    "},{"location":"elements/apps/api/","title":"Jobbergate API Overview","text":"

    The Jobbergate API is a RESTful API that functions as the Jobbergate platform's backbone. It offers access to the platform's data for various components, including the Jobbergate CLI, agent, and any other interfaces requiring interaction with Jobbergate assets.

    The API's endpoints are secured via OpenID Connect, and they require a valid auth token that is created when a user logs into the system.

    "},{"location":"elements/apps/api/#usage","title":"Usage","text":"

    The Jobbergate API is a standard RESTful API. It can be accessed vi a command-line tool like Curl or API testing tool like Postman.

    "},{"location":"elements/apps/api/#getting-an-auth-token","title":"Getting an Auth Token","text":"

    To use the Jobbergate API, you need to obtain an access token first. This token both authenticates your requests and provides authorization according to your user's rights.

    The authentication for Jobbergate API is managed by an affiliated OIDC service. While it's possible to directly interface with this service from your application to get an authentication token, the simplest method is via the Jobbergate CLI. Refer to the \"Logging In\" segment on the CLI usage page.

    "},{"location":"elements/apps/api/#querying-the-api","title":"Querying the API","text":"

    Once you have an auth token, you can interact with any of the Jobbergate API endpoints. The complete set of endpoints, parameters, and constraints are available through swagger documentation under jobbergate/docs wherever the API is deployed.

    For all requests made to secured endpoints, you must include the auth token in the Authorization header of your requests with a \"Bearer\" prefix.

    "},{"location":"elements/apps/api/#query-examples","title":"Query Examples","text":"

    To demonstrate how to use the API, the following examples will show how to fetch a list of all available Job Scripts.

    For these examples:

    • The auth token will be \"XXXXXXXX\"
    • The Jobbergate API is deployed at \"http://jobbergate.local\"
    "},{"location":"elements/apps/api/#curl","title":"curl","text":"

    From a linux terminal, you can use the curl command to make a request to the API:

    curl --header \"Authorization: Bearer XXXXXXXX\"  http://jobbergate.local/jobbergate/job-scripts\n

    The output of the above command should look something like: 

    {\"results\":[{\"id\":1,\"created_at\":\"2022-09-09T21:34:16.889289\",\"updated_at\":\"2022-09-09T21:34:16.889289\",\"job_script_name\":\"test script\",\"job_script_description\":null,\"job_script_owner_email\":\"local-user@jobbergate.local\",\"application_id\":1}],\"pagination\":{\"total\":1,\"start\":null,\"limit\":null}}\n

    To see the result more clearly, you can use a tool like jq to format the JSON response:

    {\n  \"results\": [\n    {\n      \"id\": 1,\n      \"created_at\": \"2022-09-09T21:34:16.889289\",\n      \"updated_at\": \"2022-09-09T21:34:16.889289\",\n      \"job_script_name\": \"test script\",\n      \"job_script_description\": null,\n      \"job_script_owner_email\": \"local-user@jobbergate.local\",\n      \"application_id\": 1\n    }\n  ],\n  \"pagination\": {\n    \"total\": 1,\n    \"start\": null,\n    \"limit\": null\n  }\n}\n
    "},{"location":"elements/apps/api/#python-and-httpx","title":"Python and httpx","text":"

    In python, you can use the httpx package to send requests to and process responses from the API:

    import json\nimport httpx\n\n\ntoken = \"XXXXXXXX\"\n\nresp = httpx.get(\n    \"http://localhost:8000/jobbergate/job-scripts\",\n    headers=dict(Authorization=f\"Bearer {token}\"),\n)\n\nprint(json.dumps(resp.json(), indent=2))\n

    The script will print out results like this:

    {\n  \"results\": [\n    {\n      \"id\": 1,\n      \"created_at\": \"2022-09-09T21:34:16.889289\",\n      \"updated_at\": \"2022-09-09T21:34:16.889289\",\n      \"job_script_name\": \"foo\",\n      \"job_script_description\": null,\n      \"job_script_owner_email\": \"local-user@jobbergate.local\",\n      \"application_id\": 1\n    }\n  ],\n  \"pagination\": {\n    \"total\": 1,\n    \"start\": null,\n    \"limit\": null\n  }\n}\n
    "},{"location":"elements/apps/cli/","title":"Jobbergate CLI Overview","text":"

    The Jobbergate CLI offers an interactive gateway to the functionalities of the Jobbergate API's. Users can utilize the CLI to manage resources and execute various tasks.

    The CLI operates under two primary modes:

    • Resource Creation: The CLI introduces create subcommands for every resource, allowing users to establish new instances.
    • Resource Viewing: With list and get-one subcommands available for each resource, users can inspect different detail levels about the resource entities stored in the database.

    To ensure secure access, the Jobbergate CLI offers a sign-in mechanism to the Jobbergate API. Once authenticated, users may use all the resources in Jobbergate that their account has been granted access to.

    "},{"location":"elements/apps/cli/#discovering-command-details","title":"Discovering Command details","text":"

    You can start learning about the commands and usage of the Jobbergate CLI by starting with this command:

    $ jobbergate --help\nUsage: jobbergate [OPTIONS] COMMAND [ARGS]...\n\n  Welcome to the Jobbergate CLI!\n\n  More information can be shown for each command listed below by running it\n  with the --help option.\n\nOptions:\n  --verbose / --no-verbose        Enable verbose logging to the terminal\n                                  [default: no-verbose]\n  --full / --no-full              Print all fields from CRUD commands\n                                  [default: no-full]\n  --raw / --no-raw                Print output from CRUD commands as raw json\n                                  [default: no-raw]\n  --version / --no-version        Print the version of jobbergate-cli and exit\n                                  [default: no-version]\n  --install-completion [bash|zsh|fish|powershell|pwsh]\n                                  Install completion for the specified shell.\n  --show-completion [bash|zsh|fish|powershell|pwsh]\n                                  Show completion for the specified shell, to\n                                  copy it or customize the installation.\n  --help                          Show this message and exit.\n\nCommands:\n  applications     Commands to interact with applications\n  job-scripts      Commands to interact with job scripts\n  job-submissions  Commands to interact with job submissions\n  login            Log in to the jobbergate-cli by storing the supplied...\n  logout           Logs out of the jobbergate-cli.\n  show-token       Show the token for the logged in user.\n

    If you want to delve deeper and understand the usage of a specific subcommand, you can use the --help flag with that particular subcommand. For example, to better understand the usage of the job-scripts create subcommand, you would run:

    $ jobbergate job-scripts create --help\n
    "},{"location":"elements/apps/cli/#logging-in","title":"Logging In","text":"

    The first thing you need to do with the Jobbergate CLI is to log in:

    jobbergate login\n
    Upon executing the command, a message will appear like:

    \u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Waiting for login \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502                                                                                                                      \u2502\n\u2502   To complete login, please open the following link in a browser:                                                    \u2502\n\u2502                                                                                                                      \u2502\n\u2502     http://keycloak.local:8080/realms/jobbergate-local/device?user_code=BMVJ-NLZS                                    \u2502\n\u2502                                                                                                                      \u2502\n\u2502   Waiting up to 5.0 minutes for you to complete the process...                                                       \u2502\n\u2502                                                                                                                      \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n\nWaiting for web login... \u2501\u257a\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501   3% 0:04:50\n

    Next, you will need to:

    1. Open the provided link by either clicking on it (if your terminal supports it) or copy/paste it into a browser.
    2. Enter your login credentials
    3. Complete the sign in process
    4. Return to your terminal

    You should see a message like:

    \u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Logged in! \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502                                                                                                                      \u2502\n\u2502   User was logged in with email 'local-user@jobbergate.local'                                                        \u2502\n\u2502                                                                                                                      \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n
    "},{"location":"elements/apps/cli/#checking-the-auth-token","title":"Checking the Auth Token","text":"

    To get access to the auth token you acquired by logging in, run this command:

    jobbergate show-token --plain\n
    Executing this command will display the authentication token in a plain text format, without any additional characters or formatting. This makes it easier for you to manually select and copy the token, especially in environments where clipboard access might be restricted, such as when using docker-compose or an SSH connection.

    Once the token is displayed, you can copy the token to your clipboard to use with API requests.

    It's essential to treat this token with care, as it provides access to the Jobbergate system under your user account. Ensure you don't share it with unauthorized individuals and avoid unintentionally exposing it in logs or scripts.

    "},{"location":"elements/apps/cli/#resource-commands","title":"Resource Commands","text":"

    Now that you are logged in, you can interact with any of the three main Jobbergate resources. Most of the resources provide the following sub-commands:

    • create: Create a new instance of the resource
    • delete: Delete an instance of the resource
    • get-one: Fetch details about a single instance of the resource
    • list: Fetch a listing of all the resources limited by filters
    • update: Update an instance of the resource.

    Details for each subcommand can be viewed by passing the --help flag to any of them.

    Use the --help option to explore the CLI and disccover the usage and options for all the subcommands.

    "},{"location":"elements/apps/cli/#usability","title":"Usability","text":"

    When rendering a job-script from a template, the user will be asked a series of questions to fill in the template variables.

    The library used for the questionnaire has a limitation that messages can only be displayed in a single line. This means that some of the questions can be truncated and will not be fully visible if the message is too long.

    Note

    To ensure that you can see the full output of the CLI, we recommend that you use a terminal in a maximized window.

    "},{"location":"elements/apps/core/","title":"Jobbergate Core Overview","text":"

    Jobbergate-core is an SDK designed to offer seamless access to Jobbergate's features within any Python project. It is ideal for constructing automation atop the Jobbergate platform or tailoring workflows that hinge on Jobbergate components.

    Coming Soon: More in-depth documentation of Jobbergate Core

    "},{"location":"elements/resources/","title":"Jobbergate Resources","text":"

    Jobbergate utilizes three primary resources for the efficient management of job script creation, templating, and submission. These resources are maintained in distinct database tables and can be accessed via individual API endpoints or through specific subcommands in the CLI.

    The principal resources of Jobbergate include:

    • Job Scripts
    • Job Script Templates
    • Job Submissions
    "},{"location":"elements/resources/job_script_templates/","title":"Job Script Templates","text":"

    Job Script Templates serve as adaptable blueprints for Job Scripts, allowing for the dynamic replacement of crucial values upon rendering. The end result of this process is a Job Script primed for cluster submission.

    The specific values incorporated into the template to generate a Job Script are termed \"template variables.\" Users can define constrains and default settings for these values within the Job Script Template's workflow script.

    Additionally, Job Script Templates provide a framework that allows for the interactive collection of values from users via the Jobbergate CLI.

    "},{"location":"elements/resources/job_scripts/","title":"Job Scripts","text":"

    In Jobbergate, the primary resource is the Job Script. These scripts dictate the instructions for jobs intended to execute on the Slurm cluster. They can either be Python files or shell scripts. Jobbergate facilitates the generation, modification, and submission of these Job Scripts to the cluster.

    Job Scripts can either be uploaded directly from a user's workstation or be derived by rendering the Job Script Templates.

    Submission of Job Scripts to any affiliated Slurm cluster can be accomplished through the CLI, API, or Core integrations. After submission, the execution status of a Job Script can be monitored using the Job Submission resource.

    "},{"location":"elements/resources/job_submissions/","title":"Job Submissions","text":""},{"location":"elements/resources/job_submissions/#job-submissions","title":"Job Submissions","text":"

    Job Submissions primarily monitor the status and metadata of a Job Script dispatched by Jobbergate to a Slurm cluster. They possess identifying details linking them to the Job Script that was submitted and to the corresponding Job objects created by Slurm.

    "},{"location":"reference/agent/","title":"Jobbergate Agent Reference","text":""},{"location":"reference/agent/#jobbergate_agent","title":"jobbergate_agent","text":""},{"location":"reference/agent/#jobbergate_agent.clients","title":"clients","text":""},{"location":"reference/agent/#jobbergate_agent.clients.cluster_api","title":"cluster_api","text":"

    Core module for Jobbergate API clients management

    "},{"location":"reference/agent/#jobbergate_agent.clients.cluster_api.AsyncBackendClient","title":"AsyncBackendClient","text":"

    Bases: AsyncClient

    Extends the httpx.AsyncClient class with automatic token acquisition for requests. The token is acquired lazily on the first httpx request issued. This client should be used for most agent actions.

    "},{"location":"reference/agent/#jobbergate_agent.clients.cluster_api.acquire_token","title":"acquire_token","text":"
    acquire_token(token: Token) -> Token\n

    Retrieves a token from OIDC based on the app settings.

    "},{"location":"reference/agent/#jobbergate_agent.clients.slurmrestd","title":"slurmrestd","text":"

    Core module for Jobbergate API clients management

    "},{"location":"reference/agent/#jobbergate_agent.clients.slurmrestd.AsyncBackendClient","title":"AsyncBackendClient","text":"

    Bases: AsyncClient

    Extends the httpx.AsyncClient class with automatic token acquisition for requests. The token is acquired lazily on the first httpx request issued. This client should be used for most agent actions.

    "},{"location":"reference/agent/#jobbergate_agent.clients.slurmrestd.acquire_token","title":"acquire_token","text":"
    acquire_token(username: str) -> str\n

    Retrieves a token from Slurmrestd based on the app settings.

    "},{"location":"reference/agent/#jobbergate_agent.clients.slurmrestd.inject_token","title":"inject_token","text":"
    inject_token(\n    request: httpx.Request,\n    username: typing.Optional[str] = None,\n) -> httpx.Request\n

    Inject a token based on the provided username into the request.

    For requests that need to use something except the default username, this injector should be used at the request level (instead of at client initialization) like this:

    .. code-block:: python

    client.get(url, auth=lambda r: inject_token(r, username=username))

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate","title":"jobbergate","text":""},{"location":"reference/agent/#jobbergate_agent.jobbergate.api","title":"api","text":""},{"location":"reference/agent/#jobbergate_agent.jobbergate.api.SubmissionNotifier","title":"SubmissionNotifier dataclass","text":"

    Class used to update the status for a job submission when some error is detected.

    It is designed to work together with py-buzz, extracting the error message, logging it and sending it to Jobbergate API.

    report_error async 
    report_error(params: DoExceptParams) -> None\n

    Update the status for a job submission.

    :param DoExceptParams params: Dataclass for the do_except user supplied handling method.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.api.fetch_active_submissions","title":"fetch_active_submissions async","text":"
    fetch_active_submissions() -> list[ActiveJobSubmission]\n

    Retrieve a list of active job_submissions.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.api.fetch_pending_submissions","title":"fetch_pending_submissions async","text":"
    fetch_pending_submissions() -> list[PendingJobSubmission]\n

    Retrieve a list of pending job_submissions.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.api.mark_as_submitted","title":"mark_as_submitted async","text":"
    mark_as_submitted(\n    job_submission_id: int, slurm_job_id: int\n)\n

    Mark job_submission as submitted in the Jobbergate API.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.api.update_status","title":"update_status async","text":"
    update_status(\n    job_submission_id: int,\n    status: JobSubmissionStatus,\n    *,\n    report_message: str | None = None\n) -> None\n

    Update a job submission with a status

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.constants","title":"constants","text":""},{"location":"reference/agent/#jobbergate_agent.jobbergate.constants.FileType","title":"FileType","text":"

    Bases: str, Enum

    File type enum.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.constants.JobSubmissionStatus","title":"JobSubmissionStatus","text":"

    Bases: str, Enum

    Enumeration of possible job_submission statuses.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.finish","title":"finish","text":""},{"location":"reference/agent/#jobbergate_agent.jobbergate.finish.finish_active_jobs","title":"finish_active_jobs async","text":"
    finish_active_jobs()\n

    Mark all active jobs that have completed or failed as finished.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.schemas","title":"schemas","text":""},{"location":"reference/agent/#jobbergate_agent.jobbergate.schemas.ActiveJobSubmission","title":"ActiveJobSubmission","text":"

    Bases: BaseModel

    Specialized model for the cluster-agent to pull an active job_submission.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.schemas.JobScript","title":"JobScript","text":"

    Bases: BaseModel

    Model to match database for the JobScript resource.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.schemas.JobScriptFile","title":"JobScriptFile","text":"

    Bases: BaseModel

    Model for the job_script_files field of the JobScript resource.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.schemas.PendingJobSubmission","title":"PendingJobSubmission","text":"

    Bases: BaseModel

    Specialized model for the cluster-agent to pull a pending job_submission along with data from its job_script and application sources.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.schemas.SlurmJobParams","title":"SlurmJobParams","text":"

    Bases: BaseModel

    Specialized model for describing job submission parameters for Slurm REST API.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.schemas.SlurmJobSubmission","title":"SlurmJobSubmission","text":"

    Bases: BaseModel

    Specialized model for describing a request to submit a job to Slurm REST API.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.schemas.SlurmSubmitError","title":"SlurmSubmitError","text":"

    Bases: BaseModel

    Specialized model for error content in a SlurmSubmitResponse.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.schemas.SlurmSubmitResponse","title":"SlurmSubmitResponse","text":"

    Bases: BaseModel

    Specialized model for the cluster-agent to pull a pending job_submission along with data from its job_script and application sources.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.schemas.SlurmSubmittedJobStatus","title":"SlurmSubmittedJobStatus","text":"

    Bases: BaseModel

    Specialized model for the cluster-agent to pull a concluded job_submission.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.submit","title":"submit","text":""},{"location":"reference/agent/#jobbergate_agent.jobbergate.submit.get_job_parameters","title":"get_job_parameters","text":"
    get_job_parameters(\n    slurm_parameters: Dict[str, Any], **kwargs\n) -> SlurmJobParams\n

    Obtain the job parameters from the slurm_parameters dict and additional values.

    Extra keyword arguments can be used to supply default values for any parameter (like name or current_working_directory). Note they may be overwritten by values from slurm_parameters.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.submit.submit_job_script","title":"submit_job_script async","text":"
    submit_job_script(\n    pending_job_submission: PendingJobSubmission,\n    user_mapper: SlurmUserMapper,\n) -> int\n

    Submit a Job Script to slurm via the Slurm REST API.

    :param: pending_job_submission: A job_submission with fields needed to submit. :returns: The slurm_job_id for the submitted job

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.submit.submit_pending_jobs","title":"submit_pending_jobs async","text":"
    submit_pending_jobs()\n

    Submit all pending jobs and update them with SUBMITTED status and slurm_job_id.

    :returns: The slurm_job_id for the submitted job

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.submit.unpack_error_from_slurm_response","title":"unpack_error_from_slurm_response","text":"
    unpack_error_from_slurm_response(\n    response: SlurmSubmitResponse,\n) -> str\n

    Unpack the error message from the response of a slurmrestd request.

    "},{"location":"reference/agent/#jobbergate_agent.main","title":"main","text":""},{"location":"reference/agent/#jobbergate_agent.settings","title":"settings","text":""},{"location":"reference/agent/#jobbergate_agent.settings.Settings","title":"Settings","text":"

    Bases: BaseSettings

    "},{"location":"reference/agent/#jobbergate_agent.settings.Settings.Config","title":"Config","text":"

    Provide configuration for the project settings.

    Note that we disable use of dotenv if we are in test mode.

    "},{"location":"reference/agent/#jobbergate_agent.settings.Settings.compute_extra_settings","title":"compute_extra_settings","text":"
    compute_extra_settings(values)\n

    Compute settings values that are based on other settings values.

    "},{"location":"reference/agent/#jobbergate_agent.tasks","title":"tasks","text":"

    Task definitions for the Jobbergate Agent.

    "},{"location":"reference/agent/#jobbergate_agent.tasks.active_submissions_task","title":"active_submissions_task","text":"
    active_submissions_task(scheduler: BaseScheduler) -> Job\n

    Schedule a task to handle active jobs every TASK_JOBS_INTERVAL_SECONDS seconds.

    "},{"location":"reference/agent/#jobbergate_agent.tasks.garbage_collection_task","title":"garbage_collection_task","text":"
    garbage_collection_task(\n    scheduler: BaseScheduler,\n) -> Union[Job, None]\n

    Schedule a task to perform garbage collection every dat at.

    "},{"location":"reference/agent/#jobbergate_agent.tasks.pending_submissions_task","title":"pending_submissions_task","text":"
    pending_submissions_task(scheduler: BaseScheduler) -> Job\n

    Schedule a task to submit pending jobs every TASK_JOBS_INTERVAL_SECONDS seconds.

    "},{"location":"reference/agent/#jobbergate_agent.tasks.trigger_garbage_collections","title":"trigger_garbage_collections async","text":"
    trigger_garbage_collections(\n    interval_between_calls: int = 60,\n) -> None\n

    Trigger maintenance tasks on the Jobbergate API.

    "},{"location":"reference/agent/#jobbergate_agent.utils","title":"utils","text":""},{"location":"reference/agent/#jobbergate_agent.utils.exception","title":"exception","text":"

    Core module for exception related operations

    "},{"location":"reference/agent/#jobbergate_agent.utils.exception.AuthTokenError","title":"AuthTokenError","text":"

    Bases: ClusterAgentError

    Raise exception when there are connection issues with the backend

    "},{"location":"reference/agent/#jobbergate_agent.utils.exception.ClusterAgentError","title":"ClusterAgentError","text":"

    Bases: Buzz

    Raise exception when execution command returns an error

    "},{"location":"reference/agent/#jobbergate_agent.utils.exception.JobSubmissionError","title":"JobSubmissionError","text":"

    Bases: ClusterAgentError

    Raise exception when a job cannot be submitted raises any error

    "},{"location":"reference/agent/#jobbergate_agent.utils.exception.JobbergateApiError","title":"JobbergateApiError","text":"

    Bases: ClusterAgentError

    Raise exception when communication with Jobbergate API fails

    "},{"location":"reference/agent/#jobbergate_agent.utils.exception.ProcessExecutionError","title":"ProcessExecutionError","text":"

    Bases: ClusterAgentError

    Raise exception when execution command returns an error

    "},{"location":"reference/agent/#jobbergate_agent.utils.exception.SlurmParameterParserError","title":"SlurmParameterParserError","text":"

    Bases: ClusterAgentError

    Raise exception when Slurm mapper or SBATCH parser face any error

    "},{"location":"reference/agent/#jobbergate_agent.utils.exception.SlurmrestdError","title":"SlurmrestdError","text":"

    Bases: ClusterAgentError

    Raise exception when slurmrestd raises any error

    "},{"location":"reference/agent/#jobbergate_agent.utils.exception.handle_errors_async","title":"handle_errors_async async","text":"
    handle_errors_async(\n    message: str,\n    raise_exc_class: Union[\n        Type[Exception], None\n    ] = Exception,\n    raise_args: Optional[Iterable[Any]] = None,\n    raise_kwargs: Optional[Mapping[str, Any]] = None,\n    handle_exc_class: Union[\n        Type[Exception], Tuple[Type[Exception], ...]\n    ] = Exception,\n    do_finally: Callable[[], None] = noop,\n    do_except: Callable[[DoExceptParams], None] = noop,\n    do_else: Callable[[], None] = noop,\n) -> Iterator[None]\n

    Async context manager that will intercept exceptions and repackage them with a message attached.

    Example:

    .. code-block:: python

    with handle_errors(\"It didn't work\"): some_code_that_might_raise_an_exception()

    :param: message: The message to attach to the raised exception. :param: raise_exc_class: The exception type to raise with the constructed message if an exception is caught in the managed context.

                           Defaults to Exception.\n\n                       If ``None`` is passed, no new exception will be raised and only the\n                       ``do_except``, ``do_else``, and ``do_finally``\n                       functions will be called.\n

    :param: raise_args: Additional positional args (after the constructed message) that will passed when raising an instance of the raise_exc_class. :param: raise_kwargs: Keyword args that will be passed when raising an instance of the raise_exc_class. :param: handle_exc_class: Limits the class of exceptions that will be intercepted Any other exception types will not be caught and re-packaged. Defaults to Exception (will handle all exceptions). May also be provided as a tuple of multiple exception types to handle. :param: do_finally: A function that should always be called at the end of the block. Should take no parameters. :param: do_except: A function that should be called only if there was an exception. Must accept one parameter that is an instance of the DoExceptParams dataclass. Note that the do_except method is passed the original exception. :param: do_else: A function that should be called only if there were no exceptions encountered.

    "},{"location":"reference/agent/#jobbergate_agent.utils.logging","title":"logging","text":"

    Core module for logging operations

    "},{"location":"reference/agent/#jobbergate_agent.utils.logging.log_error","title":"log_error","text":"
    log_error(params: DoExceptParams)\n

    Provide a utility function to log a Buzz-based exception and the stack-trace of the error's context.

    :param: params: A DoExceptParams instance containing the original exception, a message describing it, and the stack trace of the error.

    "},{"location":"reference/agent/#jobbergate_agent.utils.logging.logger_wraps","title":"logger_wraps","text":"
    logger_wraps(\n    *,\n    entry: bool = True,\n    exit: bool = True,\n    level: str = \"DEBUG\"\n)\n

    Decorator to wrap a function with logging statements.

    Reference

    https://loguru.readthedocs.io/en/stable/resources/recipes.html

    "},{"location":"reference/agent/#jobbergate_agent.utils.plugin","title":"plugin","text":"

    Provide to the agent the ability to load custom plugins that are installed on the same environment.

    "},{"location":"reference/agent/#jobbergate_agent.utils.plugin.load_plugins","title":"load_plugins","text":"
    load_plugins(plugin_name: str) -> Dict[str, Any]\n

    Discover and load plugins available to the agent, allowing for third party ones to be included.

    Notice the ones shipped with the agent are also declared on the pyproject.toml file as plugins, even though they could be easily loaded directly from source. This aims to support tests and to demonstrate how to use the plugin system.

    Reference

    https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins/

    "},{"location":"reference/agent/#jobbergate_agent.utils.scheduler","title":"scheduler","text":"

    Provide the task scheduler for the agent and the main loop to run it.

    Custom tasks can be added to the agent as installable plugins, which are discovered at runtime.

    References

    https://github.com/agronholm/apscheduler https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins

    "},{"location":"reference/agent/#jobbergate_agent.utils.scheduler.JobbergateTask","title":"JobbergateTask","text":"

    Bases: Protocol

    Protocol to be implemented by any task that is expected to run on the scheduler.

    __call__ 
    __call__(scheduler: BaseScheduler) -> Union[Job, None]\n

    Specify a callable used to schedule a task and return the resulting job.

    This is handled to client code to give them the opportunity to handle their own configuration and to access the rich flexibility of the scheduler API.

    None can also be returned if no task is going to be scheduled due to internal business logic.

    "},{"location":"reference/agent/#jobbergate_agent.utils.scheduler.init_scheduler","title":"init_scheduler","text":"
    init_scheduler() -> BaseScheduler\n

    Initialize the scheduler and schedule all tasks.

    "},{"location":"reference/agent/#jobbergate_agent.utils.scheduler.schedule_tasks","title":"schedule_tasks","text":"
    schedule_tasks(scheduler: BaseScheduler) -> None\n

    Discovery and schedule all tasks to be run by the agent.

    "},{"location":"reference/agent/#jobbergate_agent.utils.scheduler.shut_down_scheduler","title":"shut_down_scheduler","text":"
    shut_down_scheduler(\n    scheduler: BaseScheduler, wait: bool = True\n) -> None\n

    Shutdown the scheduler.

    "},{"location":"reference/agent/#jobbergate_agent.utils.user_mapper","title":"user_mapper","text":"

    Provide to the agent a way to map email addresses from Jobbergate local Slurm users.

    Custom mappers can be added to the agent as installable plugins, which are discovered at runtime.

    "},{"location":"reference/agent/#jobbergate_agent.utils.user_mapper.SlurmUserMapper","title":"SlurmUserMapper module-attribute","text":"
    SlurmUserMapper = Mapping[str, str]\n

    Slurm user mappers are mappings from email addresses to local Slurm users.

    "},{"location":"reference/agent/#jobbergate_agent.utils.user_mapper.SingleUserMapper","title":"SingleUserMapper dataclass","text":"

    Bases: Mapping

    A user mapper that always returns the same user.

    __post_init__ 
    __post_init__()\n

    Validate the user mapper by asserting it is not an empty string.

    "},{"location":"reference/agent/#jobbergate_agent.utils.user_mapper.SlurmUserMapperFactory","title":"SlurmUserMapperFactory","text":"

    Bases: Protocol

    Protocol to be implemented by plugins on client code.

    A callable with no arguments is expected in order to handle to client code the configuration and initialization of any custom user mapper. Any object that implements the Mapping protocol can be returned.

    __call__ 
    __call__() -> SlurmUserMapper\n

    Specify the signature to build a user mapper.

    "},{"location":"reference/agent/#jobbergate_agent.utils.user_mapper.manufacture","title":"manufacture","text":"
    manufacture() -> SlurmUserMapper\n

    Create an instance of a Slurm user mapper given the app configuration.

    "},{"location":"reference/api/","title":"Jobbergate API Reference","text":""},{"location":"reference/api/#jobbergate_api","title":"jobbergate_api","text":"

    Main components of the application: routers, config, main, pagination and create super user script.

    "},{"location":"reference/api/#jobbergate_api.apps","title":"apps","text":"

    Resources of the API.

    "},{"location":"reference/api/#jobbergate_api.apps.constants","title":"constants","text":"

    Constants to be shared by all models.

    "},{"location":"reference/api/#jobbergate_api.apps.constants.FileType","title":"FileType","text":"

    Bases: str, Enum

    File type enum.

    "},{"location":"reference/api/#jobbergate_api.apps.dependencies","title":"dependencies","text":"

    Router dependencies shared for multiple resources.

    Note

    The dependencies can be reused multiple times, since FastAPI caches the results.

    "},{"location":"reference/api/#jobbergate_api.apps.dependencies.CrudServices","title":"CrudServices","text":"

    Bases: NamedTuple

    Provide a container class for the CRUD services.

    "},{"location":"reference/api/#jobbergate_api.apps.dependencies.FileServices","title":"FileServices","text":"

    Bases: NamedTuple

    Provide a container class for the file services.

    "},{"location":"reference/api/#jobbergate_api.apps.dependencies.SecureService","title":"SecureService dataclass","text":"

    Bases: SecureSession

    Dataclass to hold the secure session and the bucket.

    "},{"location":"reference/api/#jobbergate_api.apps.dependencies.Services","title":"Services","text":"

    Bases: NamedTuple

    Provide a container class for the services.

    "},{"location":"reference/api/#jobbergate_api.apps.dependencies.get_bucket_name","title":"get_bucket_name","text":"
    get_bucket_name(\n    override_bucket_name: str | None = None,\n) -> str\n

    Get the bucket name based on the environment.

    The name can be overridden when multi tenancy is enabled by passing a bucket name.

    "},{"location":"reference/api/#jobbergate_api.apps.dependencies.get_bucket_url","title":"get_bucket_url","text":"
    get_bucket_url() -> str | None\n

    Get the bucket url based on the environment.

    "},{"location":"reference/api/#jobbergate_api.apps.dependencies.s3_bucket","title":"s3_bucket async","text":"
    s3_bucket(\n    bucket_name: str, s3_url: str | None\n) -> AsyncIterator[Bucket]\n

    Create a bucket using a context manager.

    "},{"location":"reference/api/#jobbergate_api.apps.dependencies.secure_services","title":"secure_services","text":"
    secure_services(\n    *scopes: str,\n    permission_mode: PermissionMode = PermissionMode.ALL,\n    commit: bool = True,\n    ensure_email: bool = False,\n    ensure_client_id: bool = False\n)\n

    Dependency to bind database services to a secure session.

    "},{"location":"reference/api/#jobbergate_api.apps.dependencies.service_factory","title":"service_factory","text":"
    service_factory(\n    session: AsyncSession, bucket: Bucket\n) -> Iterator[Services]\n

    Create the services and bind them to a db section and s3 bucket.

    "},{"location":"reference/api/#jobbergate_api.apps.garbage_collector","title":"garbage_collector","text":"

    Delete unused files from jobbergate's file storage.

    "},{"location":"reference/api/#jobbergate_api.apps.garbage_collector.delete_files_from_bucket","title":"delete_files_from_bucket async","text":"
    delete_files_from_bucket(\n    bucket, files_to_delete: set[str]\n) -> None\n

    Delete files from the bucket.

    "},{"location":"reference/api/#jobbergate_api.apps.garbage_collector.garbage_collect","title":"garbage_collect async","text":"
    garbage_collect(\n    session,\n    bucket,\n    list_of_tables,\n    background_tasks: BackgroundTasks,\n) -> None\n

    Delete unused files from jobbergate's file storage.

    "},{"location":"reference/api/#jobbergate_api.apps.garbage_collector.get_files_to_delete","title":"get_files_to_delete async","text":"
    get_files_to_delete(session, table, bucket) -> set[str]\n

    Get a set of files to delete.

    "},{"location":"reference/api/#jobbergate_api.apps.garbage_collector.get_set_of_files_from_bucket","title":"get_set_of_files_from_bucket async","text":"
    get_set_of_files_from_bucket(bucket, table) -> set[str]\n

    Get a set of files from the bucket.

    "},{"location":"reference/api/#jobbergate_api.apps.garbage_collector.get_set_of_files_from_database","title":"get_set_of_files_from_database async","text":"
    get_set_of_files_from_database(session, table) -> set[str]\n

    Get a set of files from the database.

    "},{"location":"reference/api/#jobbergate_api.apps.job_script_templates","title":"job_script_templates","text":"

    Module for the job script templates.

    "},{"location":"reference/api/#jobbergate_api.apps.job_script_templates.constants","title":"constants","text":"

    Describe constants for the job script templates module.

    "},{"location":"reference/api/#jobbergate_api.apps.job_script_templates.models","title":"models","text":"

    Database models for the job_script_templates resource.

    JobScriptTemplate

    Bases: CrudMixin, Base

    Job script template table definition.

    Notice all relationships are lazy=\"raise\" to prevent n+1 implicit queries. This means that the relationships must be explicitly eager loaded using helper functions in the class.

    Attributes:

    Name Type Description identifier Mapped[Optional[str]]

    The identifier of the job script template.

    template_vars Mapped[dict[str, Any]]

    The template variables of the job script template.

    See Mixin class definitions for other columns.

    include_files classmethod 
    include_files(query: Select) -> Select\n

    Include custom options on a query to eager load files.

    searchable_fields classmethod 
    searchable_fields()\n

    Add identifier as a searchable field.

    sortable_fields classmethod 
    sortable_fields()\n

    Add identifier as a sortable field.

    JobScriptTemplateFile

    Bases: FileMixin, Base

    Job script template files table definition.

    Attributes:

    Name Type Description parent_id Mapped[int]

    A foreign key to the parent job script template row.

    file_type Mapped[FileType]

    The type of the file.

    See Mixin class definitions for other columns

    WorkflowFile

    Bases: FileMixin, Base

    Workflow file table definition.

    Attributes:

    Name Type Description parent_id Mapped[int]

    A foreign key to the parent job script template row.

    runtime_config Mapped[dict[str, Any]]

    The runtime configuration of the workflow.

    See Mixin class definitions for other columns

    "},{"location":"reference/api/#jobbergate_api.apps.job_script_templates.routers","title":"routers","text":"

    Router for the Job Script Template resource.

    job_script_template_create async 
    job_script_template_create(\n    create_request: JobTemplateCreateRequest,\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_EDIT,\n            ensure_email=True,\n        )\n    ),\n)\n

    Create a new job script template.

    job_script_template_delete async 
    job_script_template_delete(\n    id_or_identifier: int | str = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_EDIT,\n            ensure_email=True,\n        )\n    ),\n)\n

    Delete a job script template by id or identifier.

    job_script_template_delete_file async 
    job_script_template_delete_file(\n    id_or_identifier: int | str = Path(),\n    file_name: str = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_EDIT,\n            ensure_email=True,\n        )\n    ),\n)\n

    Delete a file from a job script template by id or identifier.

    job_script_template_garbage_collector async 
    job_script_template_garbage_collector(\n    background_tasks: BackgroundTasks,\n    secure_services: SecureService = Depends(\n        secure_services(Permissions.JOB_TEMPLATES_EDIT)\n    ),\n)\n

    Delete all unused files from jobbergate templates on the file storage.

    job_script_template_get async 
    job_script_template_get(\n    id_or_identifier: int | str = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_VIEW, commit=False\n        )\n    ),\n)\n

    Get a job script template by id or identifier.

    job_script_template_get_file async 
    job_script_template_get_file(\n    id_or_identifier: int | str = Path(),\n    file_name: str = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_VIEW, commit=False\n        )\n    ),\n)\n

    Get a job script template file by id or identifier.

    Note

    See https://fastapi.tiangolo.com/advanced/custom-response/#streamingresponse

    job_script_template_get_list async 
    job_script_template_get_list(\n    list_params: ListParams = Depends(),\n    include_null_identifier: bool = Query(False),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_VIEW, commit=False\n        )\n    ),\n)\n

    Get a list of job script templates.

    job_script_template_update async 
    job_script_template_update(\n    update_request: JobTemplateUpdateRequest,\n    id_or_identifier: int | str = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_EDIT,\n            ensure_email=True,\n        )\n    ),\n)\n

    Update a job script template by id or identifier.

    job_script_template_upload_file async 
    job_script_template_upload_file(\n    id_or_identifier: int | str = Path(),\n    file_type: FileType = Path(),\n    upload_file: UploadFile = File(\n        ..., description=\"File to upload\"\n    ),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_EDIT,\n            ensure_email=True,\n        )\n    ),\n)\n

    Upload a file to a job script template by id or identifier.

    job_script_workflow_delete_file async 
    job_script_workflow_delete_file(\n    id_or_identifier: int | str = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_EDIT,\n            ensure_email=True,\n        )\n    ),\n)\n

    Delete a workflow file from a job script template by id or identifier.

    job_script_workflow_get_file async 
    job_script_workflow_get_file(\n    id_or_identifier: int | str = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_VIEW, commit=False\n        )\n    ),\n)\n

    Get a workflow file by id or identifier.

    Note

    See https://fastapi.tiangolo.com/advanced/custom-response/#streamingresponse

    job_script_workflow_upload_file async 
    job_script_workflow_upload_file(\n    id_or_identifier: int | str = Path(),\n    runtime_config: RunTimeConfig\n    | None = Body(\n        None,\n        description=\"Runtime configuration is optional when the workflow file already exists\",\n    ),\n    upload_file: UploadFile = File(\n        ..., description=\"File to upload\"\n    ),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_EDIT,\n            ensure_email=True,\n        )\n    ),\n)\n

    Upload a file to a job script workflow by id or identifier.

    "},{"location":"reference/api/#jobbergate_api.apps.job_script_templates.schemas","title":"schemas","text":"

    Provide schemas for the job script templates component.

    JobTemplateCreateRequest

    Bases: BaseModel

    Schema for the request to create a job template.

    JobTemplateDetailedView

    Bases: JobTemplateListView

    Schema for the request to an entry.

    Notice the files default to None, as they are not always requested, to differentiate between an empty list when they are requested, but no file is found.

    JobTemplateListView

    Bases: TableResource

    Schema for the response to get a list of entries.

    JobTemplateUpdateRequest

    Bases: BaseModel

    Schema for the request to update a job template.

    RunTimeConfig

    Bases: BaseModel

    Schema for the runtime config of a job template.

    Notice this includes user supplied variables, so it has no predefined field. It also loads the contend directly from the json at the request payload.

    __get_validators__ classmethod 
    __get_validators__()\n

    Get the validators.

    validate_to_json classmethod 
    validate_to_json(value)\n

    Validate the produced json.

    TemplateFileDetailedView

    Bases: BaseModel

    Schema for the response to get a template file.

    WorkflowFileDetailedView

    Bases: BaseModel

    Schema for the response to get a workflow file.

    "},{"location":"reference/api/#jobbergate_api.apps.job_script_templates.services","title":"services","text":"

    Services for the job_script_templates resource, including module specific business logic.

    JobScriptTemplateFileService

    Bases: FileService

    Provide an empty derived class of FileService.

    Although it doesn't do anything, it fixes errors with mypy: error: Value of type variable \"FileModel\" of \"FileService\" cannot be \"JobScriptTemplateFile\" error: Value of type variable \"FileModel\" of \"FileService\" cannot be \"WorkflowFile\"

    JobScriptTemplateService

    Bases: CrudService

    Provide a CrudService that overloads the list query builder and locator logic.

    build_list_query 
    build_list_query(\n    sort_ascending: bool = True,\n    search: str | None = None,\n    sort_field: str | None = None,\n    include_archived: bool = True,\n    include_files: bool = False,\n    include_parent: bool = False,\n    include_null_identifier: bool = True,\n    **additional_filters\n) -> Select\n

    List all job_script_templates.

    create async 
    create(**incoming_data) -> CrudModel\n

    Add a new row for the model to the database.

    locate_where_clause 
    locate_where_clause(id_or_identifier: Any) -> Any\n

    Locate an instance using the id or identifier field.

    update async 
    update(locator: Any, **incoming_data) -> CrudModel\n

    Update a row by locator with supplied data.

    validate_identifier 
    validate_identifier(identifier: str | None) -> None\n

    Validate that the identifier is not an empty string nor composed only by digits.

    Raise a ServiceError with status code 422 if the validation fails.

    Many of the job-script-template endpoints use the id or identifier interchangeably as a path parameter. With that, we need to ensure that the identifier is not a number, as that would be identified as id.

    "},{"location":"reference/api/#jobbergate_api.apps.job_scripts","title":"job_scripts","text":"

    Provide module for job_scripts.

    "},{"location":"reference/api/#jobbergate_api.apps.job_scripts.models","title":"models","text":"

    Database model for the JobScript resource.

    JobScript

    Bases: CrudMixin, Base

    Job script table definition.

    Notice all relationships are lazy=\"raise\" to prevent n+1 implicit queries. This means that the relationships must be explicitly eager loaded using helper functions in the class.

    Attributes:

    Name Type Description parent_template_id Mapped[int]

    The id of the parent template.

    See Mixin class definitions for other columns.

    include_files classmethod 
    include_files(query: Select) -> Select\n

    Include custom options on a query to eager load files.

    include_parent classmethod 
    include_parent(query: Select) -> Select\n

    Include custom options on a query to eager load parent data.

    sortable_fields classmethod 
    sortable_fields()\n

    Add parent_template_id as a sortable field.

    JobScriptFile

    Bases: FileMixin, Base

    Job script files table definition.

    Attributes:

    Name Type Description parent_template_id

    The id of the parent template.

    file_type Mapped[FileType]

    The type of the file.

    See Mixin class definitions for other columns

    "},{"location":"reference/api/#jobbergate_api.apps.job_scripts.routers","title":"routers","text":"

    Router for the Job Script Template resource.

    job_script_auto_clean_unused_entries 
    job_script_auto_clean_unused_entries(\n    background_tasks: BackgroundTasks,\n    secure_services: SecureService = Depends(\n        secure_services(Permissions.JOB_SCRIPTS_EDIT)\n    ),\n)\n

    Automatically clean unused job scripts depending on a threshold.

    job_script_create async 
    job_script_create(\n    create_request: JobScriptCreateRequest,\n    secure_services: SecureService = Depends(\n        secure_services(Permissions.JOB_SCRIPTS_EDIT)\n    ),\n)\n

    Create a stand alone job script.

    job_script_create_from_template async 
    job_script_create_from_template(\n    create_request: JobScriptCreateRequest,\n    render_request: RenderFromTemplateRequest,\n    id_or_identifier: int | str = Path(...),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SCRIPTS_EDIT, ensure_email=True\n        )\n    ),\n)\n

    Create a new job script from a job script template.

    job_script_delete async 
    job_script_delete(\n    id: int = Path(...),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SCRIPTS_EDIT, ensure_email=True\n        )\n    ),\n)\n

    Delete a job script template by id or identifier.

    job_script_delete_file async 
    job_script_delete_file(\n    id: int = Path(...),\n    file_name: str = Path(...),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SCRIPTS_EDIT, ensure_email=True\n        )\n    ),\n)\n

    Delete a file from a job script template by id or identifier.

    job_script_garbage_collector 
    job_script_garbage_collector(\n    background_tasks: BackgroundTasks,\n    secure_services: SecureService = Depends(\n        secure_services(Permissions.JOB_SCRIPTS_EDIT)\n    ),\n)\n

    Delete all unused files from job scripts on the file storage.

    job_script_get async 
    job_script_get(\n    id: int = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SCRIPTS_VIEW, commit=False\n        )\n    ),\n)\n

    Get a job script by id.

    job_script_get_file async 
    job_script_get_file(\n    id: int = Path(...),\n    file_name: str = Path(...),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SCRIPTS_VIEW, commit=False\n        )\n    ),\n)\n

    Get a job script file.

    Note

    See https://fastapi.tiangolo.com/advanced/custom-response/#streamingresponse

    job_script_get_list async 
    job_script_get_list(\n    list_params: ListParams = Depends(),\n    from_job_script_template_id: int\n    | None = Query(\n        None,\n        description=\"Filter job-scripts by the job-script-template-id they were created from.\",\n    ),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SCRIPTS_VIEW, commit=False\n        )\n    ),\n)\n

    Get a list of job scripts.

    job_script_update async 
    job_script_update(\n    update_params: JobScriptUpdateRequest,\n    id: int = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SCRIPTS_EDIT, ensure_email=True\n        )\n    ),\n)\n

    Update a job script template by id or identifier.

    job_script_upload_file async 
    job_script_upload_file(\n    id: int = Path(...),\n    file_type: FileType = Path(...),\n    upload_file: UploadFile = File(\n        ..., description=\"File to upload\"\n    ),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SCRIPTS_EDIT, ensure_email=True\n        )\n    ),\n)\n

    Upload a file to a job script.

    "},{"location":"reference/api/#jobbergate_api.apps.job_scripts.schemas","title":"schemas","text":"

    JobScript resource schema.

    JobScriptCreateRequest

    Bases: BaseModel

    Request model for creating JobScript instances.

    JobScriptDetailedView

    Bases: JobScriptListView

    Model to match database for the JobScript resource.

    JobScriptFileDetailedView

    Bases: BaseModel

    Model for the job_script_files field of the JobScript resource.

    JobScriptListView

    Bases: TableResource

    Model to match database for the JobScript resource.

    JobScriptUpdateRequest

    Bases: BaseModel

    Request model for updating JobScript instances.

    RenderFromTemplateRequest

    Bases: BaseModel

    Request model for creating a JobScript entry from a template.

    "},{"location":"reference/api/#jobbergate_api.apps.job_scripts.services","title":"services","text":"

    Services for the job_scripts resource, including module specific business logic.

    AutoCleanResponse

    Bases: NamedTuple

    Named tuple for the response of auto_clean_unused_job_scripts.

    JobScriptCrudService

    Bases: CrudService

    Provide an empty derived class of CrudService.

    Although it doesn't do anything, it fixes an error with mypy: error: Value of type variable \"CrudModel\" of \"CrudService\" cannot be \"JobScript\"

    auto_clean_unused_job_scripts async 
    auto_clean_unused_job_scripts() -> AutoCleanResponse\n

    Automatically clean unused job scripts depending on a threshold.

    Based on the last time each job script was updated or used to create a job submission, this will archived job scripts that were unarchived and delete jos script that were archived.

    delete async 
    delete(locator: Any) -> None\n

    Extend delete a row by locator.

    Orphaned job-scripts are now allowed on Jobbergate. However, the agent relies on them to submit jobs after requesting GET /agent/pending. This creates a race condition and errors occur when a job-script is deleted before the agent handles its submissions.

    To avoid this, they are marked as reject in this scenario.

    JobScriptFileService

    Bases: FileService

    Provide an empty derived class of FileService.

    Although it doesn't do anything, it fixes an error with mypy: error: Value of type variable \"FileModel\" of \"FileService\" cannot be \"JobScriptFile\"

    upsert async 
    upsert(\n    parent_id: int,\n    filename: str,\n    upload_content: str | bytes | UploadFile,\n    **upsert_kwargs\n) -> FileModel\n

    Upsert a file instance.

    validate_entrypoint_file async 
    validate_entrypoint_file(parent_id: int, filename: str)\n

    Validate that the entrypoint file is unique.

    "},{"location":"reference/api/#jobbergate_api.apps.job_scripts.tools","title":"tools","text":"

    Provide a convenience class for managing job-script files.

    inject_sbatch_params 
    inject_sbatch_params(\n    job_script_data_as_string: str, sbatch_params: list[str]\n) -> str\n

    Inject sbatch params into job script.

    Given the job script as job_script_data_as_string, inject the sbatch params in the correct location.

    "},{"location":"reference/api/#jobbergate_api.apps.job_submissions","title":"job_submissions","text":"

    Provide module for job_submissions.

    "},{"location":"reference/api/#jobbergate_api.apps.job_submissions.constants","title":"constants","text":"

    Describe constants for the job_submissions module.

    JobSubmissionStatus

    Bases: str, Enum

    Defines the set of possible statuses for a Job Submission.

    pretty_list classmethod 
    pretty_list()\n

    Return a comma-separated list of possible statuses.

    "},{"location":"reference/api/#jobbergate_api.apps.job_submissions.models","title":"models","text":"

    Database model for the JobSubmission resource.

    JobSubmission

    Bases: CrudMixin, Base

    Job submission table definition.

    Notice all relationships are lazy=\"raise\" to prevent n+1 implicit queries. This means that the relationships must be explicitly eager loaded using helper functions in the class.

    Attributes:

    Name Type Description job_script_id Mapped[int]

    Id number of the job scrip this submissions is based on.

    execution_directory Mapped[str]

    The directory where the job is executed.

    slurm_job_id Mapped[int]

    The id of the job in the slurm queue.

    client_id Mapped[str]

    The id of the custer this submission runs on.

    status Mapped[JobSubmissionStatus]

    The status of the job submission.

    report_message Mapped[str]

    The message returned by the job.

    execution_parameters Mapped[dict[str, Any]]

    The properties of the job.

    See Mixin class definitions for other columns

    include_files classmethod 
    include_files(query: Select) -> Select\n

    Include custom options on a query to eager load files.

    include_parent classmethod 
    include_parent(query: Select) -> Select\n

    Include custom options on a query to eager load parent data.

    searchable_fields classmethod 
    searchable_fields()\n

    Add client_id as a searchable field.

    sortable_fields classmethod 
    sortable_fields()\n

    Add additional sortable fields.

    "},{"location":"reference/api/#jobbergate_api.apps.job_submissions.properties_parser","title":"properties_parser","text":"

    Parser for Slurm REST API parameters from SBATCH parameters at the job script file.

    ArgumentParserCustomExit

    Bases: ArgumentParser

    Custom implementation of the built-in class for argument parsing.

    The sys.exit triggered by the original code is replaced by a ValueError, besides some friendly logging messages.

    exit 
    exit(status=0, message=None)\n

    Raise ValueError when parsing invalid parameters or if the type of their values is not correct.

    SbatchToSlurm dataclass

    Store the information for each parameter, including its name at Slurm API and SBATCH.

    Besides that, any extra argument this parameter needs when added to the parser. This information is used to build the jobscript/SBATCH parser and the two-way mapping between Slurm API and SBATCH names.

    build_mapping_sbatch_to_slurm 
    build_mapping_sbatch_to_slurm() -> bidict\n

    Create a mapper to translate in both ways between the names expected by Slurm REST API and SBATCH.

    build_parser 
    build_parser() -> ArgumentParser\n

    Build an ArgumentParser to handle all SBATCH parameters declared at sbatch_to_slurm.

    convert_sbatch_to_slurm_api 
    convert_sbatch_to_slurm_api(\n    input: Dict[str, Any]\n) -> Dict[str, Any]\n

    Take a dictionary containing key-value pairing of SBATCH parameter name space to Slurm API namespace.

    Notice the values should not be affected.

    Raise KeyError if any of the keys are unknown to the mapper.

    get_job_parameters 
    get_job_parameters(jobscript: str) -> Dict[str, Any]\n

    Parse all SBATCH parameters from a job script, map their names to Slurm API parameters.

    They are returned as a key-value pairing dictionary.

    get_job_properties_from_job_script 
    get_job_properties_from_job_script(\n    main_file_content: str, **kwargs\n) -> JobProperties\n

    Get the job properties for Slurm REST API from a job script file.

    Extra keyword arguments can be used to overwrite any parameter from the job script, like name or current_working_directory.

    jobscript_to_dict 
    jobscript_to_dict(\n    jobscript: str,\n) -> Dict[str, Union[str, bool]]\n

    Extract the SBATCH params from a given job script.

    It returns them in a dictionary for mapping the parameter names to their values.

    Raise ValueError if any of the parameters are unknown to the parser.

    "},{"location":"reference/api/#jobbergate_api.apps.job_submissions.routers","title":"routers","text":"

    Router for the JobSubmission resource.

    job_submission_agent_update async 
    job_submission_agent_update(\n    update_params: JobSubmissionAgentUpdateRequest,\n    id: int = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SUBMISSIONS_EDIT,\n            ensure_client_id=True,\n        )\n    ),\n)\n

    Update a job_submission with a new status.

    Make a put request to this endpoint with the new status to update a job_submission.

    job_submission_create async 
    job_submission_create(\n    create_request: JobSubmissionCreateRequest,\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SUBMISSIONS_EDIT,\n            ensure_email=True,\n        )\n    ),\n)\n

    Create a new job submission.

    Make a post request to this endpoint with the required values to create a new job submission.

    job_submission_delete async 
    job_submission_delete(\n    id: int = Path(\n        ...,\n        description=\"id of the job submission to delete\",\n    ),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SUBMISSIONS_EDIT,\n            ensure_email=True,\n        )\n    ),\n)\n

    Delete job_submission given its id.

    job_submission_get async 
    job_submission_get(\n    id: int = Path(...),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SUBMISSIONS_VIEW, commit=False\n        )\n    ),\n)\n

    Return the job_submission given it's id.

    job_submission_get_list async 
    job_submission_get_list(\n    list_params: ListParams = Depends(),\n    slurm_job_ids: str\n    | None = Query(\n        None,\n        description=\"Comma-separated list of slurm-job-ids to match active job_submissions\",\n    ),\n    submit_status: JobSubmissionStatus\n    | None = Query(\n        None,\n        description=\"Limit results to those with matching status\",\n    ),\n    from_job_script_id: int\n    | None = Query(\n        None,\n        description=\"Filter job-submissions by the job-script-id they were created from.\",\n    ),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SUBMISSIONS_VIEW, commit=False\n        )\n    ),\n)\n

    List job_submissions for the authenticated user.

    job_submission_update async 
    job_submission_update(\n    update_params: JobSubmissionUpdateRequest,\n    id: int = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SUBMISSIONS_EDIT,\n            ensure_email=True,\n        )\n    ),\n)\n

    Update a job_submission given its id.

    job_submissions_agent_active async 
    job_submissions_agent_active(\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SUBMISSIONS_VIEW,\n            commit=False,\n            ensure_client_id=True,\n        )\n    )\n)\n

    Get a list of active job submissions for the cluster-agent.

    job_submissions_agent_pending async 
    job_submissions_agent_pending(\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SUBMISSIONS_VIEW,\n            commit=False,\n            ensure_client_id=True,\n        )\n    )\n)\n

    Get a list of pending job submissions for the cluster-agent.

    "},{"location":"reference/api/#jobbergate_api.apps.job_submissions.schemas","title":"schemas","text":"

    JobSubmission resource schema.

    ActiveJobSubmission

    Bases: BaseModel

    Specialized model for the cluster-agent to pull an active job_submission.

    JobProperties

    Bases: BaseModel

    Specialized model for job properties.

    See more details at: https://slurm.schedmd.com/rest_api.html

    JobSubmissionAgentUpdateRequest

    Bases: BaseModel

    Request model for updating JobSubmission instances.

    JobSubmissionCreateRequest

    Bases: BaseModel

    Request model for creating JobSubmission instances.

    JobSubmissionDetailedView

    Bases: JobSubmissionListView

    Complete model to match the database for the JobSubmission resource.

    JobSubmissionListView

    Bases: TableResource

    Partial model to match the database for the JobSubmission resource.

    JobSubmissionUpdateRequest

    Bases: BaseModel

    Request model for updating JobSubmission instances.

    PendingJobSubmission

    Bases: BaseModel

    Specialized model for the cluster-agent to pull pending job_submissions.

    Model also includes data from its job_script and application sources.

    "},{"location":"reference/api/#jobbergate_api.apps.job_submissions.services","title":"services","text":"

    Services for the job_submissions resource, including module specific business logic.

    JobSubmissionService

    Bases: CrudService

    Provide a CrudService that overloads the list query builder.

    build_list_query 
    build_list_query(\n    sort_ascending: bool = True,\n    search: str | None = None,\n    sort_field: str | None = None,\n    include_archived: bool = True,\n    include_files: bool = False,\n    include_parent: bool = False,\n    filter_slurm_job_ids: list[int] | None = None,\n    **additional_filters\n) -> Select\n

    List all job_script_templates.

    "},{"location":"reference/api/#jobbergate_api.apps.models","title":"models","text":"

    Functionalities to be shared by all models.

    "},{"location":"reference/api/#jobbergate_api.apps.models.ArchiveMixin","title":"ArchiveMixin","text":"

    Add is_archived column to a table.

    Attributes:

    Name Type Description is_archived Mapped[bool]

    Specify is a row is considered archived, hidden it by default when listing rows.

    "},{"location":"reference/api/#jobbergate_api.apps.models.Base","title":"Base","text":"

    Bases: DeclarativeBase

    Base class for all models.

    References

    https://docs.sqlalchemy.org/en/20/orm/declarative_mixins.html

    "},{"location":"reference/api/#jobbergate_api.apps.models.CommonMixin","title":"CommonMixin","text":"

    Provide a dynamic table and helper methods for displaying instances.

    __str__ 
    __str__()\n

    Produce a pretty string representation of the class instance.

    __tablename__ classmethod 
    __tablename__() -> str\n

    Dynamically create table name based on the class name.

    "},{"location":"reference/api/#jobbergate_api.apps.models.CrudMixin","title":"CrudMixin","text":"

    Bases: CommonMixin, IdMixin, TimestampMixin, OwnerMixin, NameMixin, ArchiveMixin

    Add needed columns and declared attributes for all models that support a CrudService.

    include_files classmethod 
    include_files(query: Select) -> Select\n

    Include custom options on a query to eager load files.

    This should be overridden by derived classes.

    include_parent classmethod 
    include_parent(query: Select) -> Select\n

    Include custom options on a query to eager load parent data.

    This should be overridden by derived classes.

    searchable_fields classmethod 
    searchable_fields()\n

    Describe the fields that may be used in search queries.

    sortable_fields classmethod 
    sortable_fields()\n

    Describe the fields that may be used for sorting queries.

    "},{"location":"reference/api/#jobbergate_api.apps.models.FileMixin","title":"FileMixin","text":"

    Bases: CommonMixin, TimestampMixin

    Add needed columns and declared attributes for all models that support a FileService.

    Attributes:

    Name Type Description parent_id Mapped[int]

    The id of the parent row in another table. Note: Derived classes should override this attribute to make it a foreign key as well.

    description Mapped[int]

    The description of the job script template.

    file_key 
    file_key() -> str\n

    Dynamically define the s3 key for the file.

    "},{"location":"reference/api/#jobbergate_api.apps.models.IdMixin","title":"IdMixin","text":"

    Provide an id primary_key column.

    Attributes:

    Name Type Description id Mapped[int]

    The id of the job script template.

    "},{"location":"reference/api/#jobbergate_api.apps.models.NameMixin","title":"NameMixin","text":"

    Add name and description columns to a table.

    Attributes:

    Name Type Description name Mapped[str]

    The name of the job script template.

    description Mapped[str | None]

    The description of the job script template.

    "},{"location":"reference/api/#jobbergate_api.apps.models.OwnerMixin","title":"OwnerMixin","text":"

    Add an owner email columns to a table.

    Attributes:

    Name Type Description owner_email Mapped[str]

    The email of the owner of the job script template.

    "},{"location":"reference/api/#jobbergate_api.apps.models.TimestampMixin","title":"TimestampMixin","text":"

    Add timestamp columns to a table.

    Attributes:

    Name Type Description created_at Mapped[DateTime]

    The date and time when the job script template was created.

    updated_at Mapped[DateTime]

    The date and time when the job script template was updated.

    "},{"location":"reference/api/#jobbergate_api.apps.permissions","title":"permissions","text":"

    Provide a module that describes permissions in the API.

    "},{"location":"reference/api/#jobbergate_api.apps.permissions.Permissions","title":"Permissions","text":"

    Bases: str, Enum

    Describe the permissions that may be used for protecting Jobbergate routes.

    "},{"location":"reference/api/#jobbergate_api.apps.schemas","title":"schemas","text":"

    Define app-wide, reusable pydantic schemas.

    "},{"location":"reference/api/#jobbergate_api.apps.schemas.IgnoreLazyGetterDict","title":"IgnoreLazyGetterDict","text":"

    Bases: GetterDict

    A custom GetterDict to avoid triggering lazy-loads when accessing attributes.

    In this way, only explicitly joined relationships will be loaded and included in the response.

    References

    https://github.com/tiangolo/fastapi/discussions/5942

    __getitem__ 
    __getitem__(key: str) -> Any\n

    Customize getitem to avoid triggering lazy-loads when accessing attributes.

    get 
    get(key: Any, default: Any = None) -> Any\n

    Get an attribute value from the object, or return a default value if the attribute does not exist.

    "},{"location":"reference/api/#jobbergate_api.apps.schemas.ListParams","title":"ListParams","text":"

    Bases: BaseModel

    Describe the shared parameters for a list request.

    "},{"location":"reference/api/#jobbergate_api.apps.schemas.TableResource","title":"TableResource","text":"

    Bases: BaseModel

    Describes a base for table models that include basic, common info.

    "},{"location":"reference/api/#jobbergate_api.apps.services","title":"services","text":"

    Provide a generic services for CRUD and file operations in routers.

    "},{"location":"reference/api/#jobbergate_api.apps.services.BucketBoundService","title":"BucketBoundService","text":"

    Provide base class for services that bind to an s3 bucket.

    This class holds a reference to the bucket and provides methods to bind and unbind the bucket. It also keeps track of all instances of the service so that they can be iterated over.

    bucket property 
    bucket: Bucket\n

    Fetch the currently bound bucket.

    Raise an exception if the service is not bound to a bucket.

    __init__ 
    __init__()\n

    Initialize the service with a null bucket.

    bind_bucket 
    bind_bucket(bucket: Bucket)\n

    Bind the service to a bucket.

    bound_bucket 
    bound_bucket(bucket: Bucket)\n

    Provide a context within which the service is bound to a bucket.

    unbind_bucket 
    unbind_bucket()\n

    Unbind the service from a bucket.

    "},{"location":"reference/api/#jobbergate_api.apps.services.CrudModelProto","title":"CrudModelProto","text":"

    Bases: Protocol

    Provide a protocol for models that can be operated on by the CrudService.

    This protocol enables type hints for editors and type checking with mypy.

    These services would best be served by an intersection type so that the model_type is actually specified to inherit from both the mixins and the Base. This would allow static type checkers to recognize that all of the columns in a mixin are available and that the class can be instantiated in the create method. However, intersection types are not supported yet. For more information, see this discussion: https://github.com/python/typing/issues/213

    __init__ 
    __init__(**kwargs)\n

    Declare that the protocol can be instantiated.

    __tablename__ 
    __tablename__() -> str\n

    Declare that the protocol has a method to dynamically produce the table name.

    include_files classmethod 
    include_files(query: Select) -> Select\n

    Declare that the protocol has a method to include files in a query.

    include_parent classmethod 
    include_parent(query: Select) -> Select\n

    Declare that the protocol has a method to include details about the parent entry in a query.

    searchable_fields classmethod 
    searchable_fields() -> set[str]\n

    Declare that the protocol has searchable fields.

    sortable_fields classmethod 
    sortable_fields() -> set[str]\n

    Declare that the protocol has sortable fields.

    "},{"location":"reference/api/#jobbergate_api.apps.services.CrudService","title":"CrudService","text":"

    Bases: DatabaseBoundService, Generic[CrudModel]

    Provide a service that can perform various crud operations using a supplied ORM model type.

    name property 
    name\n

    Helper property to recover the name of the table.

    __init__ 
    __init__(model_type: type[CrudModel])\n

    Initialize the instance with an ORM model type.

    build_list_query 
    build_list_query(\n    sort_ascending: bool = True,\n    search: str | None = None,\n    sort_field: str | None = None,\n    include_archived: bool = True,\n    include_files: bool = False,\n    include_parent: bool = False,\n    **additional_filters\n) -> Select\n

    Build the query to list matching rows.

    Decomposed into a separate function so that deriving subclasses can add additional logic into the query.

    count async 
    count() -> int\n

    Count the number of rows in the table on the database.

    create async 
    create(**incoming_data) -> CrudModel\n

    Add a new row for the model to the database.

    delete async 
    delete(locator: Any) -> None\n

    Delete a row by locator.

    In almost all cases, the locator will just be an id value.

    ensure_attribute 
    ensure_attribute(instance: CrudModel, **attributes) -> None\n

    Ensure that a model instance has the specified values on key attributes.

    Raises HTTPException if the instance does not have the specified values.

    get async 
    get(\n    locator: Any,\n    include_files: bool = False,\n    include_parent: bool = False,\n    ensure_attributes: dict[str, Any] | None = None,\n) -> CrudModel\n

    Get a row by locator.

    In almost all cases, the locator will just be an id value.

    Key value pairs can be provided as ensure_attributes to assert that the key fields have the specified values. This is useful to assert email ownership of a row before modifying it, besides any other attribute.

    list async 
    list(**filter_kwargs) -> list[CrudModel]\n

    List all crud rows matching specified filters.

    For details on the supported filters, see the build_list_query() method.

    locate_where_clause 
    locate_where_clause(locator: Any) -> Any\n

    Provide the where clause expression to locate a row by locator.

    This method allows derived classes to locate by alternative identifiers, though locator is an id value in almost all cases. compound primary keys.

    paginated_list async 
    paginated_list(**filter_kwargs) -> Page[CrudModel]\n

    List all crud rows matching specified filters with pagination.

    For details on the supported filters, see the build_list_query() method.

    update async 
    update(locator: Any, **incoming_data) -> CrudModel\n

    Update a row by locator with supplied data.

    In almost all cases, the locator will just be an id value.

    "},{"location":"reference/api/#jobbergate_api.apps.services.DatabaseBoundService","title":"DatabaseBoundService","text":"

    Provide base class for services that bind to a database session.

    This class holds a reference to the session and provides methods to bind and unbind the session. It also keeps track of all instances of the service so that they can be iterated over.

    session property 
    session: AsyncSession\n

    Fetch the currently bound session.

    Raise an exception if the service is not bound to a session.

    __init__ 
    __init__()\n

    Instantiate the service with a null session.

    bind_session 
    bind_session(session: AsyncSession)\n

    Bind the service to a session.

    bound_session 
    bound_session(session: AsyncSession)\n

    Provide a context within which the service is bound to a session.

    unbind_session 
    unbind_session()\n

    Unbind the service from a session.

    "},{"location":"reference/api/#jobbergate_api.apps.services.FileModelProto","title":"FileModelProto","text":"

    Bases: Protocol

    Provide a protocol for models that can be operated on by the FileService.

    This protocol enables type hints for editors and type checking with mypy.

    These services would best be served by an intersection type so that the model_type is actually specified to inherit from both the mixins and the Base. This would allow static type checkers to recognize that all of the columns in a mixin are available and that the class can be instantiated in the create method. However, intersection types are not supported yet. For more information, see this discussion: https://github.com/python/typing/issues/213

    __init__ 
    __init__(**kwargs)\n

    Declare that the protocol can be instantiated.

    __tablename__ 
    __tablename__() -> str\n

    Declare that the protocol has a method to dynamically produce the table name.

    "},{"location":"reference/api/#jobbergate_api.apps.services.FileService","title":"FileService","text":"

    Bases: DatabaseBoundService, BucketBoundService, Generic[FileModel]

    Proide a service that can perform various file management operations using a supplied ORM model type.

    __init__ 
    __init__(model_type: type[FileModel])\n

    Initialize the instance with an ORM model type.

    delete async 
    delete(instance: FileModel) -> None\n

    Delete a file from s3 and from the corresponding table.

    find_children async 
    find_children(parent_id: int) -> list[FileModel]\n

    Find matching instances by parent_id.

    get async 
    get(parent_id: int, filename: str) -> FileModel\n

    Get a single instances by its parent id and filename (primary keys).

    Requires that one and only one result is found.

    get_file_content async 
    get_file_content(instance: FileModel) -> bytes\n

    Get the full contents for a file entry.

    render async 
    render(\n    instance: FileModel, parameters: dict[str, Any]\n) -> str\n

    Render the file using Jinja2.

    The parameters are passed to the template as the context, and two of them are supported: * Directly as the context, for instance, if the template contains {{ foo }}. * As a data key for backward compatibility, for instance, if the template contains {{ data.foo }}.

    stream_file_content async 
    stream_file_content(instance: FileModel) -> StreamingBody\n

    Stream the content of a file using a boto3 StreamingBody.

    The StreamingBody is an async generator that can be used for a StreamingResponse in a FastAPI app.

    upsert async 
    upsert(\n    parent_id: int,\n    filename: str,\n    upload_content: str | bytes | UploadFile,\n    **upsert_kwargs\n) -> FileModel\n

    Upsert a file instance.

    "},{"location":"reference/api/#jobbergate_api.apps.services.ServiceError","title":"ServiceError","text":"

    Bases: HTTPException

    Make HTTPException more friendly by changing the default behavior so that the first arg is a message.

    Also needed to play nice with py-buzz methods.

    __init__ 
    __init__(\n    message,\n    status_code=status.HTTP_400_BAD_REQUEST,\n    **kwargs\n)\n

    Instantiate the HTTPException super class by setting detail to the message provided.

    "},{"location":"reference/api/#jobbergate_api.config","title":"config","text":"

    Provide configuration settings for the app.

    Pull settings from environment variables or a .env file if available.

    "},{"location":"reference/api/#jobbergate_api.config.LogLevelEnum","title":"LogLevelEnum","text":"

    Bases: str, Enum

    Provide an enumeration class describing the available log levels.

    "},{"location":"reference/api/#jobbergate_api.config.Settings","title":"Settings","text":"

    Bases: BaseSettings

    Provide a pydantic BaseSettings model for the application settings.

    "},{"location":"reference/api/#jobbergate_api.config.Settings.remove_blank_env","title":"remove_blank_env","text":"
    remove_blank_env(values)\n

    Remove any settings from the environment that are blank strings.

    This allows the defaults to be set if docker-compose defaults a missing environment variable to a blank string.

    "},{"location":"reference/api/#jobbergate_api.config.check_none_or_all_keys_exist","title":"check_none_or_all_keys_exist","text":"
    check_none_or_all_keys_exist(\n    input_dict: dict, target_keys: set\n) -> bool\n

    Verify if none or all of the target keys exist in the input dictionary.

    "},{"location":"reference/api/#jobbergate_api.email_notification","title":"email_notification","text":"

    Email notification system for Jobbergate.

    "},{"location":"reference/api/#jobbergate_api.email_notification.EmailManager","title":"EmailManager dataclass","text":"

    Email manager.

    "},{"location":"reference/api/#jobbergate_api.email_notification.EmailManager.send_email","title":"send_email","text":"
    send_email(\n    to_emails: Union[str, List[str]],\n    subject: str,\n    skip_on_failure: bool = False,\n    **kwargs\n) -> None\n

    Send an email using this manager.

    "},{"location":"reference/api/#jobbergate_api.email_notification.EmailNotificationError","title":"EmailNotificationError","text":"

    Bases: Buzz

    Custom error to be raised for problems at the email notification system.

    "},{"location":"reference/api/#jobbergate_api.email_notification.notify_submission_rejected","title":"notify_submission_rejected","text":"
    notify_submission_rejected(\n    job_submission_id: Union[str, int],\n    report_message: str,\n    to_emails: Union[str, List[str]],\n) -> None\n

    Notify an email or a list of emails about a job submission that has been rejected.

    "},{"location":"reference/api/#jobbergate_api.main","title":"main","text":"

    Main file to startup the fastapi server.

    "},{"location":"reference/api/#jobbergate_api.main.health_check","title":"health_check async","text":"
    health_check()\n

    Provide a health-check endpoint for the app.

    "},{"location":"reference/api/#jobbergate_api.main.lifespan","title":"lifespan async","text":"
    lifespan(_: FastAPI)\n

    Provide a lifespan context for the app.

    Will set up logging and cleanup database engines when the app is shut down.

    This is the preferred method of handling lifespan events in FastAPI. For mor details, see: https://fastapi.tiangolo.com/advanced/events/

    "},{"location":"reference/api/#jobbergate_api.main.validation_exception_handler","title":"validation_exception_handler async","text":"
    validation_exception_handler(\n    request: Request, err: RequestValidationError\n)\n

    Handle exceptions from pydantic validators.

    "},{"location":"reference/api/#jobbergate_api.meta_mapper","title":"meta_mapper","text":"

    Provides a metadata-mapper for re-using descriptions and examples across many pydantic models.

    "},{"location":"reference/api/#jobbergate_api.meta_mapper.MetaField","title":"MetaField dataclass","text":"

    Provides a dataclass that describes the metadata that will be mapped for an individual field.

    "},{"location":"reference/api/#jobbergate_api.meta_mapper.MetaMapper","title":"MetaMapper","text":"

    Maps re-usable metadata for fields. Should be used with the schema_extra property of a Model's Config.

    Example::

    foo_meta = MetaMapper(\n    id=MetaField(\n        description=\"The unique identifier of this Foo\",\n        example=13,\n    ),\n    name=MetaField(\n        description=\"The name of this Foo\",\n        example=\"Bar\",\n    ),\n    is_active=MetaField(\n        description=\"Indicates if this Foo is active\",\n        example=True,\n    ),\n    created_at=MetaField(\n        description=\"The timestamp indicating when this Foo was created\",\n        example=\"2023-08-18T13:55:37.172285\",\n    ),\n)\n\n\nclass CreateFooRequest(BaseModel):\n    name: str\n    is_active: Optional[bool]\n\n    class Config:\n        schema_extra = foo_meta\n\n\nclass UpdateFooRequest(BaseModel):\n    name: Optional[str] = None\n    is_active: Optional[bool] = None\n\n    class Config:\n        schema_extra = foo_meta\n\n\nclass FooResponse(BaseModel):\n    id: int\n    name: str\n    is_active: bool\n    created_at: DateTime\n\n    class Config:\n        schema_extra = foo_meta\n

    Notice in this example that the fields may be required in some models and optional in others. Further, not all the fields are present in all the models. The MetaMapper allows the models to share field metadata and yet define the fields independently.

    "},{"location":"reference/api/#jobbergate_api.meta_mapper.MetaMapper.__call__","title":"__call__","text":"
    __call__(schema: Dict[str, Any], *_) -> None\n

    Map the MetaFields onto the metadata properties of a schema.

    Should be used in a pydantic Model's Config class.

    "},{"location":"reference/api/#jobbergate_api.meta_mapper.MetaMapper.__init__","title":"__init__","text":"
    __init__(**kwargs: MetaField)\n

    Map the kwargs into the field_dict.

    All kwargs should be MetaFields, but any object duck-typed to include all the attributes of a MetaField will be accepted.

    "},{"location":"reference/api/#jobbergate_api.safe_types","title":"safe_types","text":"

    Provide \"safe\" type annotatons to avoid issues with mypy and Fast api.

    Regarding the JobScript and JobSubmission type

    These are needed for the relationships in the models. This avoids issues with circular imports at runtime.

    Regarding the Bucket type

    This is necessary because the Bucket type isn't importable from the normal boto3 modules. Instead, it must be imported from the mypy typing plugin for boto3.

    The \"type\" must be bound to Any when not type checking because FastAPI does type inspection for its dependency injection system. Thus, there must be a type associated with Bucket even when not type checking.

    "},{"location":"reference/api/#jobbergate_api.security","title":"security","text":"

    Instantiates armasec resources for auth on api endpoints using project settings.

    Also provides a factory function for TokenSecurity to reduce boilerplate.

    "},{"location":"reference/api/#jobbergate_api.security.IdentityPayload","title":"IdentityPayload","text":"

    Bases: TokenPayload

    Provide an extension of TokenPayload that includes the user's identity.

    "},{"location":"reference/api/#jobbergate_api.security.IdentityPayload.extract_organization","title":"extract_organization","text":"
    extract_organization(values)\n

    Extract the organization_id from the organization payload.

    The payload is expected to look like: { ..., \"organization\": { \"adf99e01-5cd5-41ac-a1af-191381ad7780\": { ... } } }

    "},{"location":"reference/api/#jobbergate_api.security.get_domain_configs","title":"get_domain_configs","text":"
    get_domain_configs() -> list[DomainConfig]\n

    Return a list of DomainConfig objects based on the input variables for the Settings class.

    "},{"location":"reference/api/#jobbergate_api.security.lockdown_with_identity","title":"lockdown_with_identity","text":"
    lockdown_with_identity(\n    *scopes: str,\n    permission_mode: PermissionMode = PermissionMode.ALL,\n    ensure_email: bool = False,\n    ensure_organization: bool = False,\n    ensure_client_id: bool = False\n)\n

    Provide a wrapper to be used with dependency injection to extract identity on a secured route.

    "},{"location":"reference/api/#jobbergate_api.storage","title":"storage","text":"

    Provide functions to interact with persistent data storage.

    "},{"location":"reference/api/#jobbergate_api.storage.EngineFactory","title":"EngineFactory","text":"

    Provide a factory class that creates engines and keeps track of them in an engine mapping.

    This is used for multi-tenancy and database URL creation at request time.

    "},{"location":"reference/api/#jobbergate_api.storage.EngineFactory.__init__","title":"__init__","text":"
    __init__()\n

    Initialize the EngineFactory.

    "},{"location":"reference/api/#jobbergate_api.storage.EngineFactory.auto_session","title":"auto_session async","text":"
    auto_session(\n    override_db_name: str | None = None, commit: bool = True\n) -> typing.AsyncIterator[AsyncSession]\n

    Get an asynchronous database session.

    Gets a new session from the correct engine in the engine map.

    "},{"location":"reference/api/#jobbergate_api.storage.EngineFactory.cleanup","title":"cleanup async","text":"
    cleanup()\n

    Close all engines stored in the engine map and clears the engine_map.

    "},{"location":"reference/api/#jobbergate_api.storage.EngineFactory.get_engine","title":"get_engine","text":"
    get_engine(\n    override_db_name: str | None = None,\n) -> AsyncEngine\n

    Get a database engine.

    If the database url is already in the engine map, return the engine stored there. Otherwise, build a new one, store it, and return the new engine.

    "},{"location":"reference/api/#jobbergate_api.storage.SecureSession","title":"SecureSession dataclass","text":"

    Provide a container class for an IdentityPayload and AsyncSesson for the current request.

    "},{"location":"reference/api/#jobbergate_api.storage.build_db_url","title":"build_db_url","text":"
    build_db_url(\n    override_db_name: str | None = None,\n    force_test: bool = False,\n    asynchronous: bool = True,\n) -> str\n

    Build a database url based on settings.

    If force_test is set, build from the test database settings. If asynchronous is set, use asyncpg. If override_db_name replace the database name in the settings with the supplied value.

    "},{"location":"reference/api/#jobbergate_api.storage.handle_fk_error","title":"handle_fk_error","text":"
    handle_fk_error(\n    _: fastapi.Request,\n    err: asyncpg.exceptions.ForeignKeyViolationError,\n)\n

    Unpack metadata from a ForeignKeyViolationError and return a 409 response.

    "},{"location":"reference/api/#jobbergate_api.storage.render_sql","title":"render_sql","text":"
    render_sql(session: AsyncSession, query) -> str\n

    Render a sqlalchemy query into a string for debugging.

    "},{"location":"reference/api/#jobbergate_api.storage.search_clause","title":"search_clause","text":"
    search_clause(\n    search_terms: str, searchable_fields: set\n) -> ColumnElement[bool]\n

    Create search clause across searchable fields with search terms.

    Regarding the False first argument to or_(): The or_() function must have one fixed positional argument. See: https://docs.sqlalchemy.org/en/20/core/sqlelement.html#sqlalchemy.sql.expression.or_

    "},{"location":"reference/api/#jobbergate_api.storage.secure_session","title":"secure_session","text":"
    secure_session(\n    *scopes: str,\n    permission_mode: PermissionMode = PermissionMode.ALL,\n    commit: bool = True,\n    ensure_email: bool = False,\n    ensure_organization: bool = False,\n    ensure_client_id: bool = False\n)\n

    Provide an injectable for FastAPI that checks permissions and returns a database session for this request.

    This should be used for all secured routes that need access to the database. It will commit the transaction upon completion of the request. If an exception occurs, it will rollback the transaction. If multi-tenancy is enabled, it will retrieve a database session for the database associated with the client_id found in the requesting user's auth token.

    If testing mode is enabled, it will flush the session instead of committing changes to the database.

    Note that the session should NEVER be explicitly committed anywhere else in the source code.

    "},{"location":"reference/api/#jobbergate_api.storage.sort_clause","title":"sort_clause","text":"
    sort_clause(\n    sort_field: str,\n    sortable_fields: set,\n    sort_ascending: bool,\n) -> typing.Union[Mapped, UnaryExpression, Case]\n

    Create a sort clause given a sort field, the list of sortable fields, and a sort_ascending flag.

    "},{"location":"reference/api/#jobbergate_api.version","title":"version","text":"

    Provide the version of the package.

    "},{"location":"reference/api/#jobbergate_api.version.get_version","title":"get_version","text":"
    get_version() -> str\n

    Get the version from the metadata if available, otherwise from pyproject.toml.

    Returns \"unknown\" if both methods fail.

    "},{"location":"reference/api/#jobbergate_api.version.get_version_from_metadata","title":"get_version_from_metadata","text":"
    get_version_from_metadata() -> str\n

    Get the version from the metadata.

    This is the preferred method of getting the version, but only works if the package is properly installed in a Python environment.

    "},{"location":"reference/api/#jobbergate_api.version.get_version_from_poetry","title":"get_version_from_poetry","text":"
    get_version_from_poetry() -> str\n

    Get the version from pyproject.toml.

    This is a fallback method if the package is not installed, but just copied and accessed locally, like in a Docker image.

    "},{"location":"reference/cli/","title":"Jobbergate CLI Reference","text":""},{"location":"reference/cli/#jobbergate_cli","title":"jobbergate_cli","text":"

    Jobbergate command-line interface and app library

    "},{"location":"reference/cli/#jobbergate_cli.__getattr__","title":"__getattr__","text":"
    __getattr__(name: str)\n

    Overload module attribute lookup to warn if 'appform' is being imported because it is deprecated.

    "},{"location":"reference/cli/#jobbergate_cli.application_base","title":"application_base","text":"

    Provide a stub module to maintain compatibility with previous versions.

    Issue a deprecation warning when this module is imported from if JOBBERGATE_COMPATIBILITY_MODE is enabled.

    If JOBBERGATE_COMPATIBILITY_MODE is not enabled, raise an import error when this module is imported.

    "},{"location":"reference/cli/#jobbergate_cli.auth","title":"auth","text":"

    Utilities for handling auth in jobbergate-cli.

    "},{"location":"reference/cli/#jobbergate_cli.auth.clear_token_cache","title":"clear_token_cache","text":"
    clear_token_cache()\n

    Clears the token cache.

    "},{"location":"reference/cli/#jobbergate_cli.auth.fetch_auth_tokens","title":"fetch_auth_tokens","text":"
    fetch_auth_tokens(ctx: JobbergateContext) -> TokenSet\n

    Fetch an access token (and possibly a refresh token) from Auth0.

    Prints out a URL for the user to use to authenticate and polls the token endpoint to fetch it when the browser-based process finishes

    "},{"location":"reference/cli/#jobbergate_cli.auth.init_persona","title":"init_persona","text":"
    init_persona(\n    ctx: JobbergateContext,\n    token_set: Optional[TokenSet] = None,\n)\n

    Initializes the \"persona\" which contains the tokens and email address for a user.

    Retrieves the access token for the user from the cache.

    Token is retrieved from the cache, validated, and user email is extracted.

    If the access token is expired, a new one will be acquired via the cached refresh token (if there is one).

    Saves token_set to cache.

    Returns the persona.

    "},{"location":"reference/cli/#jobbergate_cli.auth.load_tokens_from_cache","title":"load_tokens_from_cache","text":"
    load_tokens_from_cache() -> TokenSet\n

    Loads an access token (and a refresh token if one exists) from the cache.

    "},{"location":"reference/cli/#jobbergate_cli.auth.open_on_browser","title":"open_on_browser","text":"
    open_on_browser(url: str) -> bool\n

    Open the url on the browser using webbrowser.

    "},{"location":"reference/cli/#jobbergate_cli.auth.refresh_access_token","title":"refresh_access_token","text":"
    refresh_access_token(\n    ctx: JobbergateContext, token_set: TokenSet\n)\n

    Attempt to fetch a new access token given a refresh token in a token_set.

    Sets the access token in-place.

    If refresh fails, notify the user that they need to log in again.

    "},{"location":"reference/cli/#jobbergate_cli.auth.save_tokens_to_cache","title":"save_tokens_to_cache","text":"
    save_tokens_to_cache(token_set: TokenSet)\n

    Saves tokens from a token_set to the cache.

    "},{"location":"reference/cli/#jobbergate_cli.auth.show_login_message","title":"show_login_message","text":"
    show_login_message(verification_uri: str)\n

    Show a message to the user with a link to the auth provider to login.

    "},{"location":"reference/cli/#jobbergate_cli.auth.validate_token_and_extract_identity","title":"validate_token_and_extract_identity","text":"
    validate_token_and_extract_identity(\n    token_set: TokenSet,\n) -> IdentityData\n

    Validate the access_token from a TokenSet and extract the user's identity data.

    Validations
    • Checks if access_token is not empty.
    • Checks timestamp on the access token.
    • Checks that the client_id is present
    • Checks that email is present

    Reports an error in the logs and to the user if there is an issue with the access_token.

    "},{"location":"reference/cli/#jobbergate_cli.compat","title":"compat","text":"

    Provide compatibility to the previous version of Jobbergate CLI for users who have automation or are familiar with the old commands

    "},{"location":"reference/cli/#jobbergate_cli.compat.add_legacy_compatible_commands","title":"add_legacy_compatible_commands","text":"
    add_legacy_compatible_commands(app: typer.Typer)\n

    Add commands from the restructured CLI under the previous names for the commands to the root typer app.

    "},{"location":"reference/cli/#jobbergate_cli.config","title":"config","text":"

    Configuration file, sets all the necessary environment variables. Can load configuration from a dotenv file if supplied.

    "},{"location":"reference/cli/#jobbergate_cli.config.Settings","title":"Settings","text":"

    Bases: BaseSettings

    Provide a pydantic settings model to hold configuration values loaded from the environment.

    "},{"location":"reference/cli/#jobbergate_cli.config.Settings.Config","title":"Config","text":"

    Customize behavior of the Settings class. Especially, enable the use of dotenv to load settings from a .env file instead of the environment.

    "},{"location":"reference/cli/#jobbergate_cli.config.Settings.compute_extra_settings","title":"compute_extra_settings","text":"
    compute_extra_settings(values)\n

    Compute settings values that are based on other settings values.

    "},{"location":"reference/cli/#jobbergate_cli.config.build_settings","title":"build_settings","text":"
    build_settings(*args, **kwargs)\n

    Return a Setting object and handle ValidationError with a message to the user.

    "},{"location":"reference/cli/#jobbergate_cli.constants","title":"constants","text":"

    Provide constants that may be used throughout the CLI modules.

    "},{"location":"reference/cli/#jobbergate_cli.constants.FileType","title":"FileType","text":"

    Bases: str, Enum

    File type enum.

    "},{"location":"reference/cli/#jobbergate_cli.constants.SortOrder","title":"SortOrder","text":"

    Bases: str, Enum

    Enum descring the type of sort orders that are available for list commands.

    "},{"location":"reference/cli/#jobbergate_cli.exceptions","title":"exceptions","text":"

    Provide exceptions and custom handlers for the CLI.

    "},{"location":"reference/cli/#jobbergate_cli.exceptions.Abort","title":"Abort","text":"

    Bases: Buzz

    A special exception used to abort the Jobbergate CLI.

    Collects information provided for use in the handle_abort context manager.

    "},{"location":"reference/cli/#jobbergate_cli.exceptions.Abort.__init__","title":"__init__","text":"
    __init__(\n    message,\n    *args,\n    subject=None,\n    support=False,\n    log_message=None,\n    sentry_context=None,\n    original_error=None,\n    warn_only=False,\n    **kwargs\n)\n

    Initialize the Abort errror.

    "},{"location":"reference/cli/#jobbergate_cli.exceptions.JobbergateCliError","title":"JobbergateCliError","text":"

    Bases: Buzz

    A generic exception base class to use in Jobbergate CLI

    "},{"location":"reference/cli/#jobbergate_cli.exceptions.handle_abort","title":"handle_abort","text":"
    handle_abort(func)\n

    Apply a decorator to gracefully handle any Abort errors that happen within the context.

    Will log the error, dispatch it to Sentry, show a helpful message to the user about the error, and exit.

    "},{"location":"reference/cli/#jobbergate_cli.jobberappslib","title":"jobberappslib","text":"

    Provide a stub module to maintain compatibility with previous versions.

    Issue a deprecation warning when this module is imported from if JOBBERGATE_COMPATIBILITY_MODE is enabled.

    If JOBBERGATE_COMPATIBILITY_MODE is not enabled, raise an import error when this module is imported.

    "},{"location":"reference/cli/#jobbergate_cli.logging","title":"logging","text":"

    Provide initializers for logging.

    "},{"location":"reference/cli/#jobbergate_cli.logging.init_logs","title":"init_logs","text":"
    init_logs(verbose=False)\n

    Initialize logging.

    If JOBBERGATE_LOG_PATH is set in the config, add a rotatating file log handler. Logs will be retained for 1 week.

    If verbose is supplied, add a stdout handler at the DEBUG level.

    "},{"location":"reference/cli/#jobbergate_cli.logging.init_sentry","title":"init_sentry","text":"
    init_sentry()\n

    Initialize Sentry if the SENTRY_DSN environment variable is present.

    "},{"location":"reference/cli/#jobbergate_cli.main","title":"main","text":"

    Provide main entry point for the Jobbergate CLI App.

    "},{"location":"reference/cli/#jobbergate_cli.main.login","title":"login","text":"
    login(ctx: typer.Context)\n

    Log in to the jobbergate-cli by storing the supplied token argument in the cache.

    "},{"location":"reference/cli/#jobbergate_cli.main.logout","title":"logout","text":"
    logout()\n

    Logs out of the jobbergate-cli. Clears the saved user credentials.

    "},{"location":"reference/cli/#jobbergate_cli.main.main","title":"main","text":"
    main(\n    ctx: typer.Context,\n    verbose: bool = typer.Option(\n        False, help=\"Enable verbose logging to the terminal\"\n    ),\n    full: bool = typer.Option(\n        False, help=\"Print all fields from CRUD commands\"\n    ),\n    raw: bool = typer.Option(\n        False,\n        help=\"Print output from CRUD commands as raw json\",\n    ),\n    version: bool = typer.Option(\n        False,\n        help=\"Print the version of jobbergate-cli and exit\",\n    ),\n    ignore_extra_args: str = typer.Option(\n        None,\n        \"--username\",\n        \"-u\",\n        \"--password\",\n        \"-p\",\n        hidden=True,\n        help=\"Ignore extra arguments passed to the command for backward compatibility with the legacy app.\",\n    ),\n)\n

    Welcome to the Jobbergate CLI!

    More information can be shown for each command listed below by running it with the --help option.

    "},{"location":"reference/cli/#jobbergate_cli.main.show_token","title":"show_token","text":"
    show_token(\n    plain: bool = typer.Option(\n        False, help=\"Show the token in plain text.\"\n    ),\n    refresh: bool = typer.Option(\n        False,\n        help=\"Show the refresh token instead of the access token.\",\n    ),\n    show_prefix: bool = typer.Option(\n        False,\n        \"--prefix\",\n        help=\"Include the 'Bearer' prefix in the output.\",\n    ),\n    show_header: bool = typer.Option(\n        False,\n        \"--header\",\n        help=\"Show the token as it would appear in a request header.\",\n    ),\n    decode: bool = typer.Option(\n        False,\n        \"--decode\",\n        help=\"Show the content of the decoded access token.\",\n    ),\n)\n

    Show the token for the logged in user.

    Token output is automatically copied to your clipboard.

    "},{"location":"reference/cli/#jobbergate_cli.render","title":"render","text":"

    Provide helpers to render output for users.

    "},{"location":"reference/cli/#jobbergate_cli.render.StyleMapper","title":"StyleMapper","text":"

    Provide a mapper that can set rich styles for rendered output of data tables and dicts.

    The subapps have list endpoints that return sets of values. These are rendered as tables in the output. The StyleMapper class provides a way to simply define styles that should be applied to the columns of the table.

    Example:

    The following code will print a table where the columns are colored according to the style_mapper

    .. code-block: python

    style_mapper = StyleMapper( a=\"bold green\", b=\"red\", c=\"blue\", ) envelope = dict( results=[ dict(a=1, b=2, c=3), dict(a=4, b=5, c=6), dict(a=7, b=8, c=9), ], pagination=dict(total=3) ) render_list_results(jb_ctx, envelope, style_mapper)

    "},{"location":"reference/cli/#jobbergate_cli.render.StyleMapper.__init__","title":"__init__","text":"
    __init__(**colors: str)\n

    Initialize the StyleMapper.

    "},{"location":"reference/cli/#jobbergate_cli.render.StyleMapper.map_style","title":"map_style","text":"
    map_style(column: str) -> Dict[str, Any]\n

    Map a column name from the table to display to the style that should be used to render it.

    "},{"location":"reference/cli/#jobbergate_cli.render.render_dict","title":"render_dict","text":"
    render_dict(\n    data: Dict[str, Any],\n    title: str = \"Data\",\n    hidden_fields: Optional[List[str]] = None,\n)\n

    Render a dictionary in a rich Table That shows the key and value of each item.

    :param: data: The dictionary to render :param: title: The title header to include above the Table output :param: hidden_fields: Keys that should be hidden in the Table output

    "},{"location":"reference/cli/#jobbergate_cli.render.render_json","title":"render_json","text":"
    render_json(data: Any)\n

    Print nicely formatted representation of a JSON serializable python primitive.

    "},{"location":"reference/cli/#jobbergate_cli.render.render_list_results","title":"render_list_results","text":"
    render_list_results(\n    ctx: JobbergateContext,\n    envelope: ListResponseEnvelope,\n    style_mapper: Optional[StyleMapper] = None,\n    hidden_fields: Optional[List[str]] = None,\n    title: str = \"Results List\",\n)\n

    Render a list of result data items in a rich Table.

    :param: ctx: The JobbergateContext. This is needed to detect if full or raw output is needed :param: envelope: A ListResponseEnvelope containing the data items :param: style_mapper: The style mapper that should be used to apply styles to the columns of the table :param: hidden_fields: Columns that should (if not using full mode) be hidden in the Table output :param: title: The title header to include above the Table output

    "},{"location":"reference/cli/#jobbergate_cli.render.render_single_result","title":"render_single_result","text":"
    render_single_result(\n    ctx: JobbergateContext,\n    result: Union[Dict[str, Any], pydantic.BaseModel],\n    hidden_fields: Optional[List[str]] = None,\n    title: str = \"Result\",\n)\n

    Render a single data item in a rich ``Table.

    :param: ctx: The JobbergateContext. This is needed to detect if full` orrawoutput is needed :param: result: The data item to display. May be a dict or a pydantic model. :param: hidden_fields: Rows that should (if not usingfullmode) be hidden in theTableoutput :param: title: The title header to include above theTale`` output

    "},{"location":"reference/cli/#jobbergate_cli.render.terminal_message","title":"terminal_message","text":"
    terminal_message(\n    message,\n    subject=None,\n    color=\"green\",\n    footer=None,\n    indent=True,\n)\n

    Print a nicely formatted message as output to the user using a rich Panel.

    :param: message: The message to print out :param: subject: An optional subject line to add in the header of the Panel :param: color: An optional color to style the subject header with :param: footer: An optional message to display in the footer of the Panel :param: indent: Adds padding to the left of the message

    "},{"location":"reference/cli/#jobbergate_cli.requests","title":"requests","text":"

    Provide utilities for making requests against the Jobbergate API.

    "},{"location":"reference/cli/#jobbergate_cli.requests.make_request","title":"make_request","text":"
    make_request(\n    client: httpx.Client,\n    url_path: str,\n    method: str,\n    *,\n    expected_status: Optional[int] = None,\n    expect_response: bool = True,\n    abort_message: str = \"There was an error communicating with the API\",\n    abort_subject: str = \"REQUEST FAILED\",\n    support: bool = True,\n    response_model_cls: Optional[\n        Type[ResponseModel]\n    ] = None,\n    request_model: Optional[pydantic.BaseModel] = None,\n    save_to_file: Optional[Path] = None,\n    **request_kwargs: Any\n) -> Union[ResponseModel, Dict, int]\n

    Make a request against the Jobbergate API.

    :param: client: The Httpx client to use for the request :param: url_path: The path to add to the base url of the client where the request should be sent :param: method: The REST method to use for the request (GET, PUT, UPDATE, POST, DELETE, etc) :param: expected_status: The status code to expect on the response. If it is not received, raise an Abort :param: expect_response: Indicates if response data (JSON) is expected from the API endpoint :param: abort_message: The message to show the user if there is a problem and the app must be aborted :param: abort_subject: The subject to use in Abort output to the user :param: support: If true, add a message to the output instructing the user to seek help :param: response_model_cls: If supplied, serialize the response data into this Pydantic model class :param: request_model: Use a pydantic model instance as the data body for the request :param: request_kwargs: Any additional keyword arguments that need to be passed on to the client

    "},{"location":"reference/cli/#jobbergate_cli.schemas","title":"schemas","text":"

    Provide Pydantic models for various data items.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.ApplicationResponse","title":"ApplicationResponse","text":"

    Bases: BaseModel

    Describes the format of data for applications retrieved from the Jobbergate API endpoints.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.ClusterCacheData","title":"ClusterCacheData","text":"

    Bases: BaseModel

    Describes the format of data stored in the clusters cache file.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.DeviceCodeData","title":"DeviceCodeData","text":"

    Bases: BaseModel

    A model representing the data that is returned from the OIDC provider's device code endpoint.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.IdentityData","title":"IdentityData","text":"

    Bases: BaseModel

    A model representing the identifying data for a user from an auth token.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobScriptCreateRequest","title":"JobScriptCreateRequest","text":"

    Bases: BaseModel

    Request model for creating JobScript instances.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobScriptFiles","title":"JobScriptFiles","text":"

    Bases: BaseModel

    Model containing job-script files.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobScriptRenderRequestData","title":"JobScriptRenderRequestData","text":"

    Bases: BaseModel

    Describes the data that will be sent to the create endpoint of the Jobbergate API for job scripts.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobScriptResponse","title":"JobScriptResponse","text":"

    Bases: BaseModel

    Describes the format of data for job_scripts retrieved from the Jobbergate API endpoints.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobScriptResponse.null_files","title":"null_files","text":"
    null_files(value)\n

    Remap a None value in files to an empty list.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobSubmissionCreateRequestData","title":"JobSubmissionCreateRequestData","text":"

    Bases: BaseModel

    Describes the data that will be sent to the create endpoint of the Jobbergate API for job submissions.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobSubmissionResponse","title":"JobSubmissionResponse","text":"

    Bases: BaseModel

    Describes the format of data for job_submissions retrieved from the Jobbergate API endpoints.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobbergateApplicationConfig","title":"JobbergateApplicationConfig","text":"

    Bases: BaseModel

    A data object describing the config data needed to instantiate a JobbergateApplication class.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobbergateConfig","title":"JobbergateConfig","text":"

    Bases: BaseModel

    A data object describing the config values needed in the \"jobbergate_config\" section of the JobbergateApplicationConfig model.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobbergateConfig.compute_extra_settings","title":"compute_extra_settings","text":"
    compute_extra_settings(values)\n

    Compute missing values and extra operations to enhance the user experience and backward compatibility.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobbergateContext","title":"JobbergateContext","text":"

    Bases: BaseModel

    A data object describing context passed from the main entry point.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.ListResponseEnvelope","title":"ListResponseEnvelope","text":"

    Bases: BaseModel

    A model describing the structure of response envelopes from \"list\" endpoints.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.Persona","title":"Persona","text":"

    Bases: BaseModel

    A model representing a pairing of a TokenSet and user email. This is a convenience to combine all of the identifying data and credentials for a given user.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.RenderFromTemplateRequest","title":"RenderFromTemplateRequest","text":"

    Bases: BaseModel

    Request model for creating a JobScript entry from a template.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.TokenSet","title":"TokenSet","text":"

    Bases: BaseModel

    A model representing a pairing of access and refresh tokens

    "},{"location":"reference/cli/#jobbergate_cli.subapps","title":"subapps","text":"

    Subapps that are added to the base Typer application.

    "},{"location":"reference/cli/#jobbergate_cli.subapps.applications","title":"applications","text":"

    Provide a sub-app for interacting with Applications data.

    "},{"location":"reference/cli/#jobbergate_cli.subapps.applications.app","title":"app","text":"

    Provide a typer app that can interact with Application data in a cruddy manner.

    create 
    create(\n    ctx: typer.Context,\n    name: str = typer.Option(\n        ...,\n        \"--name\",\n        \"-n\",\n        help=\"The name of the application to create\",\n    ),\n    identifier: Optional[str] = typer.Option(\n        None,\n        help=f\"The human-friendly identifier of the application. {IDENTIFIER_NOTE}\",\n    ),\n    application_path: pathlib.Path = typer.Option(\n        ...,\n        \"--application-path\",\n        \"-a\",\n        help=\"The path to the directory where the application files are located\",\n    ),\n    application_desc: Optional[str] = typer.Option(\n        None,\n        help=\"A helpful description of the application\",\n    ),\n)\n

    Create a new application.

    delete 
    delete(\n    ctx: typer.Context,\n    id: Optional[int] = typer.Option(\n        None,\n        \"--id\",\n        \"-i\",\n        help=f\"The specific id of the application to delete. {ID_NOTE}\",\n    ),\n    identifier: Optional[str] = typer.Option(\n        None,\n        help=f\"The human-friendly identifier of the application to update. {IDENTIFIER_NOTE}\",\n    ),\n)\n

    Delete an existing application.

    download_files 
    download_files(\n    ctx: typer.Context,\n    id: Optional[int] = typer.Option(\n        None,\n        help=f\"The specific id of the application. {ID_NOTE}\",\n    ),\n    identifier: Optional[str] = typer.Option(\n        None,\n        help=f\"The human-friendly identifier of the application. {IDENTIFIER_NOTE}\",\n    ),\n)\n

    Download the files from an application to the current working directory.

    get_one 
    get_one(\n    ctx: typer.Context,\n    id: Optional[int] = typer.Option(\n        None,\n        \"--id\",\n        \"-i\",\n        help=f\"The specific id of the application. {ID_NOTE}\",\n    ),\n    identifier: Optional[str] = typer.Option(\n        None,\n        help=f\"The human-friendly identifier of the application. {IDENTIFIER_NOTE}\",\n    ),\n)\n

    Get a single application by id or identifier

    list_all 
    list_all(\n    ctx: typer.Context,\n    show_all: bool = typer.Option(\n        False,\n        \"--all\",\n        help=\"Show all applications, even the ones without identifier\",\n    ),\n    user_only: bool = typer.Option(\n        False,\n        \"--user\",\n        help=\"Show only applications owned by the current user\",\n    ),\n    search: Optional[str] = typer.Option(\n        None, help=\"Apply a search term to results\"\n    ),\n    sort_order: SortOrder = typer.Option(\n        SortOrder.UNSORTED, help=\"Specify sort order\"\n    ),\n    sort_field: Optional[str] = typer.Option(\n        None,\n        help=\"The field by which results should be sorted\",\n    ),\n)\n

    Show available applications

    update 
    update(\n    ctx: typer.Context,\n    id: Optional[int] = typer.Option(\n        None,\n        \"--id\",\n        \"-i\",\n        help=f\"The specific id of the application to update. {ID_NOTE}\",\n    ),\n    identifier: Optional[str] = typer.Option(\n        None,\n        help=f\"The human-friendly identifier of the application to update. {IDENTIFIER_NOTE}\",\n    ),\n    application_path: Optional[pathlib.Path] = typer.Option(\n        None,\n        \"--application-path\",\n        \"-a\",\n        help=\"The path to the directory where the application files are located\",\n    ),\n    update_identifier: Optional[str] = typer.Option(\n        None,\n        help=\"Optional new application identifier to be set\",\n    ),\n    application_desc: Optional[str] = typer.Option(\n        None,\n        help=\"Optional new application description to be set\",\n    ),\n    application_name: Optional[str] = typer.Option(\n        None, help=\"Optional new application name to be set\"\n    ),\n)\n

    Update an existing application.

    "},{"location":"reference/cli/#jobbergate_cli.subapps.applications.application_base","title":"application_base","text":"

    ApplicationBase.

    JobbergateApplicationBase

    JobbergateApplicationBase.

    __init__ 
    __init__(jobbergate_yaml: Dict[str, Any])\n

    Initialize class attributes.

    find_templates staticmethod 
    find_templates(\n    application_path: pathlib.Path,\n) -> List[pathlib.Path]\n

    Finds templates a given application path.

    mainflow 
    mainflow(data: Dict[str, Any])\n

    Implements the main question asking workflow.

    "},{"location":"reference/cli/#jobbergate_cli.subapps.applications.application_helpers","title":"application_helpers","text":"

    Helper functions that may be used inside of Jobbergate applications.

    get_file_list 
    get_file_list(path=None, search_term='*.*')\n

    Return a list of input files in a directory that match a search term.

    Ignore casing when comparing against the search term.

    Default to searching for all files in the current directory.

    get_running_jobs 
    get_running_jobs(user_only=True)\n

    Return a list of the user's currently running jobs, as given by SLURM's squeue command.

    The format returned is: [job ID, 8 chars] [job name]

    "},{"location":"reference/cli/#jobbergate_cli.subapps.applications.questions","title":"questions","text":"

    Abstraction layer for questions. Each class represents different question types.

    The questions describe literal questions that are asked of the user in an interactive mode via the inquirer package.

    Questions will be skipped and use the default value if the ignore property resolves to True.

    Questions will also resolve to their default values if running in \"fast mode\".

    BooleanList

    Bases: Confirm

    Asks a confirmation question that is followed up by a certain question list when true and a different list if false.

    __init__ 
    __init__(\n    variablename: str,\n    message: str,\n    whentrue=None,\n    whenfalse=None,\n    **kwargs\n)\n

    Initialize the Checkbox question.

    :param: variablename: The key in the config dictionary that this question will set :param: message: The message to show the user that describes what the question is gathering :param whentrue: List of questions to ask if user answers 'true' on this question :param whentrue: List of questions to show if user answers 'false' on this question

    ignore_child 
    ignore_child(\n    child: QuestionBase, answers: Dict[str, Any]\n) -> bool\n

    Dynamically check if a child question should be ignored based on the questions that have already been answered.

    :param: child: The child question that might be ignored :param: answers: Answer values to previously asked questions

    make_ignore_partial 
    make_ignore_partial(\n    child: QuestionBase,\n) -> Callable[[Dict[str, Any]], bool]\n

    Build a partial method for checking if a child should be ignored.

    This method just makes the code more readable so that a non-descriptive lambda does not need to be used inline.

    make_prompts 
    make_prompts(**override_kwargs)\n

    Create inquirer prompts from this instance of BooleanList and for all its child questions.

    :param: override_kwargs: A collection of keyword arguments to override in the base make_prompts method

    Checkbox

    Bases: QuestionBase

    Gives the user a list to choose multiple entries from.

    __init__ 
    __init__(\n    variablename: str, message: str, choices: list, **kwargs\n)\n

    Initialize the Checkbox question.

    :param: variablename: The key in the config dictionary that this question will set :param: message: The message to show the user that describes what the question is gathering :param: choices: A list of the possible values from which the Question will allow the user to select many

    Confirm

    Bases: QuestionBase

    Asks a question with a boolean answer (true/false).

    __init__ 
    __init__(variablename: str, message: str, **kwargs)\n

    Initialize the Confirm question.

    :param: variablename: The key in the config dictionary that this question will set :param: message: The message to show the user that describes what the question is gathering

    Const

    Bases: Text

    Sets the variable to the default value. Doesn't show anything.

    __init__ 
    __init__(variablename: str, **kwargs)\n

    Initialize the Const \"question\".

    :param: variablename: The key in the config dictionary that this question will set

    make_prompts 
    make_prompts()\n

    Create inquirer prompts from this instance of Const.

    Directory

    Bases: QuestionBase

    Asks for a directory name. If exists is True it checks if path exists and is a directory.

    :param exists: Checks if given directory exists

    __init__ 
    __init__(\n    variablename: str,\n    message: str,\n    exists: Optional[bool] = None,\n    **kwargs\n)\n

    Initialize the Directory question.

    :param: variablename: The key in the config dictionary that this question will set :param: message: The message to show the user that describes what the question is gathering :param: exists: If True, ensure that the directory exists on the system

    File

    Bases: QuestionBase

    Asks for a file name.

    __init__ 
    __init__(\n    variablename: str,\n    message: str,\n    exists: Optional[bool] = None,\n    **kwargs\n)\n

    Initialize the File question.

    :param: variablename: The key in the config dictionary that this question will set :param: message: The message to show the user that describes what the question is gathering :param: exists: If True, ensure that the file path exists on the system

    Integer

    Bases: QuestionBase

    Asks for an integer value. Could have min and/or max constrains.

    __init__ 
    __init__(\n    variablename: str,\n    message: str,\n    minval: Optional[int] = None,\n    maxval: Optional[int] = None,\n    **kwargs\n)\n

    Initialize the Integer question.

    :param: variablename: The key in the config dictionary that this question will set :param: message: The message to show the user that describes what the question is gathering :param: minval: The minimum value the integer may be set to. If not specified, use negative infinity. :param: minval: The maximum value the integer may be set to. If not specified, use infinity.

    List

    Bases: QuestionBase

    Gives the user a list to choose one from.

    __init__ 
    __init__(\n    variablename: str, message: str, choices: list, **kwargs\n)\n

    Initialize the List question.

    :param: variablename: The key in the config dictionary that this question will set :param: message: The message to show the user that describes what the question is gathering :param: choices: A list of the possible values from which the Question will allow the user to select one

    QuestionBase

    Baseclass for questions.

    All questions have variablename, message and an optional default.

    __init__ 
    __init__(\n    variablename: str,\n    message: str,\n    ignore: bool = False,\n    default: Optional[Any] = None,\n    inquirer_type: Type[TInquirerType] = inquirer.Text,\n)\n

    Initialize the Question.

    :param: variablename: The key in the config dictionary that this question will set :param: message: The message to show the user that describes what the question is gathering :param: ignore: If true, do not ask the question and just use the default value instead :param: default: The default value for the variablename in the answers dict :param: inquirer_type: The inquirer question type that this QuestionBase wraps

    make_prompts 
    make_prompts(**override_kwargs)\n

    Create inquirer prompts from this instance of QuestionBase.

    :param: override_kwargs: A collection of keyword arguments to override in intializing the inquirer question

    Text

    Bases: QuestionBase

    Asks for a text value.

    gather_param_values 
    gather_param_values(\n    application: JobbergateApplicationBase,\n    supplied_params: Optional[Dict[str, Any]] = None,\n    fast_mode: bool = False,\n) -> Dict[str, Any]\n

    Gather the parameter values by executing the application methods.

    Prompt users for answers or use defaults as needed.

    :param: application: The application instance to pull questions from :param: supplied_params: Pre-supplied parameters. Any questions where the variablename matches a pre-supplied key in the dict at the start of execution will be skipped. :param: fast_mode: Do not ask the user questions. Just use the supplied params and defaults. :returns: A dict of the gathered parameter values

    "},{"location":"reference/cli/#jobbergate_cli.subapps.applications.tools","title":"tools","text":"

    Provide tool functions for working with Application data.

    execute_application 
    execute_application(\n    app_module: JobbergateApplicationBase,\n    app_config: JobbergateApplicationConfig,\n    supplied_params: Optional[Dict[str, Any]] = None,\n    fast_mode: bool = False,\n)\n

    Execute the jobbergate application python module.

    Updates the app_config with values gathered in the question workflow

    :param: app_module: The source code for the application to execute :param: app_config: The configuration for the JobbergateApplication :param: supplied_params: Pre-set values for the parameters. Any questions about these values will be skipped. :param: fast_mode: If true, do not ask the user questions. Just use supplied_params or defaults :returns: The configuration values collected from the user by executing the application

    fetch_application_data 
    fetch_application_data(\n    jg_ctx: JobbergateContext,\n    id: Optional[int] = None,\n    identifier: Optional[str] = None,\n) -> ApplicationResponse\n

    Retrieve an application from the API by id or identifier.

    :param: jg_ctx: The JobbergateContext. Needed to access the Httpx client with which to make API calls :param: id: The id of the application to fetch :param: identifier: If supplied, look for an application instance with the provided identifier :returns: An instance of ApplicationResponse containing the application data

    get_upload_files 
    get_upload_files(application_path: pathlib.Path)\n

    Context manager to build the files parameter.

    Open the supplied file(s) and build a files param appropriate for using multi-part file uploads with the client.

    load_application_config_from_source 
    load_application_config_from_source(\n    config_source: str,\n) -> JobbergateApplicationConfig\n

    Load the JobbergateApplicationConfig from a text string containing the config as YAML.

    :param: config_source: The YAML containing the config :returns: A JobbergateApplicationConfig instance with the config values

    load_application_data 
    load_application_data(\n    app_data: ApplicationResponse,\n    application_source_file: str,\n) -> Tuple[\n    JobbergateApplicationConfig, JobbergateApplicationBase\n]\n

    Validates and loads the data for an application returned from the API's applications GET endpoint.

    As part of the Jobbergate data restructure, sections of the legacy jobbergate.yaml are now stored in different tables in the backend. This function reconstructs them from app_data.workflow_file.runtime_config and app_data.template_vars for backward compatibility.

    :param: app_data: A dictionary containing the application data :returns: A tuple containing the application config and the application module

    load_application_from_source 
    load_application_from_source(\n    app_source: str, app_config: JobbergateApplicationConfig\n) -> JobbergateApplicationBase\n

    Load the JobbergateApplication class from a text string containing the source file.

    Creates the module in a temporary file and imports it with importlib.

    Adapted from: https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly

    :param: app_source: The JobbergateApplication source code to load :param: app_config: The JobbergateApplicationConfig needed to instantiate the JobbergateApplication

    load_default_config 
    load_default_config() -> Dict[str, Any]\n

    Load the default config for an application.

    save_application_files 
    save_application_files(\n    jg_ctx: JobbergateContext,\n    application_data: ApplicationResponse,\n    destination_path: pathlib.Path,\n) -> List[pathlib.Path]\n

    Save the application files from the API response into a local destination.

    upload_application 
    upload_application(\n    jg_ctx: JobbergateContext,\n    application_path: pathlib.Path,\n    application_id: Optional[int],\n    application_identifier: Optional[str],\n) -> bool\n

    Upload an application given an application path and the application id.

    :param: jg_ctx: The JobbergateContext. Needed to access the Httpx client with which to make API calls :param: application_path: The directory where the application files to upload may be found :param: application_id: The id of the application for which to upload data :param: application_identifier: The identifier of the application for which to upload data :returns: True if the upload was successful; False otherwise

    "},{"location":"reference/cli/#jobbergate_cli.subapps.clusters","title":"clusters","text":"

    Provide a sub-app for interacting with Cluster data.

    "},{"location":"reference/cli/#jobbergate_cli.subapps.clusters.app","title":"app","text":"

    Provide a typer app that can interact with Cluster data in a cruddy manner.

    list_all 
    list_all(ctx: typer.Context)\n

    Show available clusters

    "},{"location":"reference/cli/#jobbergate_cli.subapps.clusters.tools","title":"tools","text":"

    Provide tool functions for working with Cluster data

    "},{"location":"reference/cli/#jobbergate_cli.subapps.job_scripts","title":"job_scripts","text":"

    Provide a sub-app for interacting with Job Script data.

    "},{"location":"reference/cli/#jobbergate_cli.subapps.job_scripts.app","title":"app","text":"

    Provide a typer app that can interact with Job Script data in a cruddy manner.

    delete 
    delete(\n    ctx: typer.Context,\n    id: int = typer.Option(\n        ...,\n        \"--id\",\n        \"-i\",\n        help=\"The id of the job script to delete\",\n    ),\n)\n

    Delete an existing job script.

    download_files 
    download_files(\n    ctx: typer.Context,\n    id: int = typer.Option(\n        ..., help=\"The specific id of the job script.\"\n    ),\n)\n

    Download the files from a job script to the current working directory.

    get_one 
    get_one(\n    ctx: typer.Context,\n    id: int = typer.Option(\n        ...,\n        \"--id\",\n        \"-i\",\n        help=\"The specific id of the job script.\",\n    ),\n)\n

    Get a single job script by id.

    list_all 
    list_all(\n    ctx: typer.Context,\n    show_all: bool = typer.Option(\n        False,\n        \"--all\",\n        help=\"Show all job scripts, even the ones owned by others\",\n    ),\n    search: Optional[str] = typer.Option(\n        None, help=\"Apply a search term to results\"\n    ),\n    sort_order: SortOrder = typer.Option(\n        SortOrder.UNSORTED, help=\"Specify sort order\"\n    ),\n    sort_field: Optional[str] = typer.Option(\n        None,\n        help=\"The field by which results should be sorted\",\n    ),\n    from_application_id: Optional[int] = typer.Option(\n        None,\n        help=\"Filter job-scripts by the application-id they were created from.\",\n    ),\n)\n

    Show available job scripts

    render 
    render(\n    ctx: typer.Context,\n    name: Optional[str] = typer.Option(\n        None,\n        \"--name\",\n        \"-n\",\n        help=dedent(\n            \"\\n            The name of the job script to create.\\n            If this is not supplied, the name will be derived from the base application.\\n            \"\n        ),\n    ),\n    application_id: Optional[int] = typer.Option(\n        None,\n        \"--application-id\",\n        \"-i\",\n        help=\"The id of the application from which to create the job script.\",\n    ),\n    application_identifier: Optional[str] = typer.Option(\n        None,\n        help=\"The identifier of the application from which to create the job script.\",\n    ),\n    description: Optional[str] = typer.Option(\n        None,\n        help=\"Optional text describing the job script.\",\n    ),\n    sbatch_params: Optional[List[str]] = typer.Option(\n        None,\n        help=\"Optional parameter to submit raw sbatch parameters.\",\n    ),\n    param_file: Optional[pathlib.Path] = typer.Option(\n        None,\n        help=dedent(\n            \"\\n            Supply a json file that contains the parameters for populating templates.\\n            If this is not supplied, the question asking in the application is triggered.\\n            \"\n        ),\n    ),\n    cluster_name: Optional[str] = typer.Option(\n        None,\n        help=\"The name of the cluster where the job should be submitted (i.g. 'nash-staging')\",\n    ),\n    execution_directory: Optional[\n        pathlib.Path\n    ] = typer.Option(\n        None,\n        help=dedent(\n            '\\n            The path on the cluster where the job script should be executed.\\n            If provided as a relative path, it will be converted as an absolute path from your current\\n            working directory. If you use \"~\" to denote your home directory, the path will be expanded to an\\n            absolute path for your home directory on *this* machine.\\n            '\n        ).strip(),\n    ),\n    download: Optional[bool] = typer.Option(\n        None,\n        help=\"Download the job script files to the current working directory\",\n    ),\n    fast: bool = typer.Option(\n        False,\n        \"--fast\",\n        \"-f\",\n        help=\"Use default answers (when available) instead of asking the user.\",\n    ),\n    submit: Optional[bool] = typer.Option(\n        None,\n        help=\"Do not ask the user if they want to submit a job.\",\n    ),\n)\n

    Render a new job script from an application.

    show_files 
    show_files(\n    ctx: typer.Context,\n    id: int = typer.Option(\n        ..., help=\"The specific id of the job script.\"\n    ),\n    plain: bool = typer.Option(\n        False, help=\"Show the files in plain text.\"\n    ),\n)\n

    Show the files for a single job script by id.

    update 
    update(\n    ctx: typer.Context,\n    id: int = typer.Option(\n        ...,\n        \"--id\",\n        \"-i\",\n        help=\"The id of the job script to update\",\n    ),\n    name: Optional[str] = typer.Option(\n        None, help=\"Optional new name of the job script.\"\n    ),\n    description: Optional[str] = typer.Option(\n        None,\n        help=\"Optional new text describing the job script.\",\n    ),\n)\n

    Update an existing job script.

    "},{"location":"reference/cli/#jobbergate_cli.subapps.job_scripts.tools","title":"tools","text":"

    Provide tool functions for working with Job Script data

    download_job_script_files 
    download_job_script_files(\n    id: int, jg_ctx: JobbergateContext\n) -> List[pathlib.Path]\n

    Download the job script files from the API and save them to the current working directory.

    fetch_job_script_data 
    fetch_job_script_data(\n    jg_ctx: JobbergateContext, id: int\n) -> JobScriptResponse\n

    Retrieve a job_script from the API by id

    flatten_param_dict 
    flatten_param_dict(\n    param_dict: Dict[str, Any]\n) -> Dict[str, Any]\n

    Flatten an input dictionary to support the rendering process.

    See the example:

    param_dict = { ... \"application_config\": {\"job_name\": \"rats\", \"partitions\": [...]}, ... \"jobbergate_config\": { ... \"default_template\": \"test_job_script.sh\", ... \"supporting_files\": [...], ... \"supporting_files_output_name\": {...}, ... \"template_files\": [...], ... \"job_script_name\": None, ... \"output_directory\": \".\", ... \"partition\": \"debug\", ... \"job_name\": \"rats\", ... }, ... } flat_param_dict = flatten_param_dict(param_dict) print(flat_param_dict) { \"job_name\": \"rats\", \"partitions\": [\"debug\", \"partition1\"], \"default_template\": \"test_job_script.sh\", \"supporting_files\": [\"test_job_script.sh\"], \"supporting_files_output_name\": {\"test_job_script.sh\": [...]}, \"template_files\": [\"templates/test_job_script.sh\"], \"job_script_name\": None, \"output_directory\": \".\", \"partition\": \"debug\", }

    get_template_output_name_mapping 
    get_template_output_name_mapping(\n    config: JobbergateConfig, job_name: str\n) -> Dict[str, str]\n

    Get the mapping of template names to output names.

    This provides the mapping as expected by the API v4 from the configuration on CLI v3.

    question_helper 
    question_helper(\n    question_func: Callable,\n    text: str,\n    default: Any,\n    fast: bool,\n    actual_value: Optional[Any],\n)\n

    Helper function for asking questions to the user.

    :param Callable question_func: The function to use to ask the question :param str text: The text of the question to ask :param Any default: The default value to use if the user does not provide one :param bool fast: Whether to use default answers (when available) instead of asking the user :param Any actual_value: The actual value provided by the user, if any

    :returns: actual_value or default or the value provided by the user

    The actual_value has the most priority and will be returned if it is not None. After evaluating the actual_value, the fast mode will determine if the default value will be used. Otherwise, the question will be prompted to the user.

    remove_prefix 
    remove_prefix(s: str) -> str\n

    Remove the prefix 'templates/' from a string

    remove_prefix_suffix 
    remove_prefix_suffix(s: str) -> str\n

    Remove the prefix 'templates/' and suffixes '.j2' and '.jinja2' from a string

    render_job_script 
    render_job_script(\n    jg_ctx: JobbergateContext,\n    name: Optional[str] = None,\n    application_id: Optional[int] = None,\n    application_identifier: Optional[str] = None,\n    description: Optional[str] = None,\n    sbatch_params: Optional[List[str]] = None,\n    param_file: Optional[pathlib.Path] = None,\n    fast: bool = False,\n) -> JobScriptResponse\n

    Render a new job script from an application.

    :param str name: Name of the new job script. :param Optional[int] application_id: Id of the base application. :param Optional[str] application_identifier: Identifier of the base application. :param Optional[str] description: Description of the new job script. :param Optional[List[str]] sbatch_params: List of sbatch parameters. :param Optional[pathlib.Path] param_file: Path to a parameters file. :param bool fast: Whether to use default answers (when available) instead of asking the user. :param JobbergateContext jg_ctx: The Jobbergate context. :return JobScriptResponse: The new job script.

    save_job_script_files 
    save_job_script_files(\n    jg_ctx: JobbergateContext,\n    job_script_data: JobScriptResponse,\n    destination_path: pathlib.Path,\n) -> List[pathlib.Path]\n

    Save the job script files from the API response to the output path.

    update_template_files_information 
    update_template_files_information(\n    app_data: ApplicationResponse,\n    app_config: JobbergateApplicationConfig,\n)\n

    Update the information about the template files if not already present in the configuration.

    upload_job_script_files 
    upload_job_script_files(\n    jg_ctx: JobbergateContext,\n    job_script_id: int,\n    job_script_path: pathlib.Path,\n    supporting_file_paths: Optional[\n        List[pathlib.Path]\n    ] = None,\n)\n

    Upload a job-script and its supporting files given their paths and the job-script id.

    :param: jg_ctx: The JobbergateContext. Needed to access the Httpx client with which to make API calls :param: job_script_path: The path to the job-script file to upload :param: supporting_file_paths: The paths to any supporting files to upload with the job-scritpt :param: job_script_id: The id of the job-script for which to upload data :returns: True if the main job script upload was successful; False otherwise

    validate_parameter_file 
    validate_parameter_file(\n    parameter_path: pathlib.Path,\n) -> Dict[str, Any]\n

    Validate parameter file at the supplied path and returns the parsed dict.

    Confirms

    parameter_path exists parameter_path is a valid json file

    "},{"location":"reference/cli/#jobbergate_cli.subapps.job_submissions","title":"job_submissions","text":"

    Provide a sub-app for interacting with Job Submission data.

    "},{"location":"reference/cli/#jobbergate_cli.subapps.job_submissions.app","title":"app","text":"

    Provide a typer app that can interact with Job Submission data in a cruddy manner.

    create 
    create(\n    ctx: typer.Context,\n    name: str = typer.Option(\n        ...,\n        \"--name\",\n        \"-n\",\n        help=\"The name of the job submission to create\",\n    ),\n    description: Optional[str] = typer.Option(\n        None,\n        help=\"A helpful description of the job submission\",\n    ),\n    job_script_id: int = typer.Option(\n        ...,\n        \"--job-script-id\",\n        \"-i\",\n        help=\"The id of the job_script from which to create the job submission\",\n    ),\n    cluster_name: str = typer.Option(\n        None,\n        help=\"The name of the cluster where the job should be submitted (i.g. 'nash-staging')\",\n    ),\n    execution_directory: Optional[Path] = typer.Option(\n        None,\n        help=dedent(\n            '\\n            The path on the cluster where the job script should be executed.\\n            If provided as a relative path, it will be converted as an absolute path from your current\\n            working directory. If you use \"~\" to denote your home directory, the path will be expanded to an\\n            absolute path for your home directory on *this* machine.\\n            '\n        ).strip(),\n    ),\n    execution_parameters: Optional[Path] = typer.Option(\n        None,\n        help=dedent(\n            \"\\n            The path to a JSON file containing the parameters to be passed to the job submission.\\n            See more details at: https://slurm.schedmd.com/rest_api.html\\n            \"\n        ).strip(),\n        exists=True,\n        readable=True,\n        resolve_path=True,\n    ),\n    download: bool = typer.Option(\n        False,\n        help=\"Download the job script files to the current working directory\",\n    ),\n)\n

    Create a new job submission.

    delete 
    delete(\n    ctx: typer.Context,\n    id: int = typer.Option(\n        ...,\n        \"--id\",\n        \"-i\",\n        help=\"The id of the job submission to delete\",\n    ),\n)\n

    Delete an existing job submission.

    get_one 
    get_one(\n    ctx: typer.Context,\n    id: int = typer.Option(\n        ...,\n        \"--id\",\n        \"-i\",\n        help=\"The specific id of the job submission.\",\n    ),\n)\n

    Get a single job submission by id

    list_all 
    list_all(\n    ctx: typer.Context,\n    show_all: bool = typer.Option(\n        False,\n        \"--all\",\n        help=\"Show all job submissions, even the ones owned by others\",\n    ),\n    search: Optional[str] = typer.Option(\n        None, help=\"Apply a search term to results\"\n    ),\n    sort_order: SortOrder = typer.Option(\n        SortOrder.UNSORTED, help=\"Specify sort order\"\n    ),\n    sort_field: Optional[str] = typer.Option(\n        None,\n        help=\"The field by which results should be sorted\",\n    ),\n    from_job_script_id: Optional[int] = typer.Option(\n        None,\n        help=\"Filter job-submissions by the job-script-id they were created from.\",\n    ),\n)\n

    Show available job submissions.

    "},{"location":"reference/cli/#jobbergate_cli.subapps.job_submissions.tools","title":"tools","text":"

    Provide tool functions for working with Job Submission data

    create_job_submission 
    create_job_submission(\n    jg_ctx: JobbergateContext,\n    job_script_id: int,\n    name: str,\n    description: Optional[str] = None,\n    cluster_name: Optional[str] = None,\n    execution_directory: Optional[Path] = None,\n    execution_parameters_file: Optional[Path] = None,\n) -> JobSubmissionResponse\n

    Create a Job Submission from the given Job Script.

    :param: jg_ctx: The JobbergateContext. Used to retrieve the client for requests and the email of the submitting user :param: job_script_id: The id of the Job Script to submit to Slurm :param: name: The name to attach to the Job Submission :param: description: An optional description that may be added to the Job Submission :param: cluster_name: An optional cluster_name for the cluster where the job should be executed, If left off, it will default to the DEFAULT_CLUSTER_NAME from the settings. If no default is set, an exception will be raised. :param: execution_directory: An optional directory where the job should be executed. If provided as a relative path, it will be constructed as an absolute path relative to the current working directory. :param: execution_parameters_file: An optional file containing the execution parameters for the job.

    :returns: The Job Submission data returned by the API after creating the new Job Submission

    fetch_job_submission_data 
    fetch_job_submission_data(\n    jg_ctx: JobbergateContext, job_submission_id: int\n) -> JobSubmissionResponse\n

    Retrieve a job submission from the API by id

    "},{"location":"reference/cli/#jobbergate_cli.text_tools","title":"text_tools","text":"

    Provide some basic tools for manipulating text.

    "},{"location":"reference/cli/#jobbergate_cli.text_tools.conjoin","title":"conjoin","text":"
    conjoin(*items: str, join_str: str = '\\n') -> str\n

    Joins strings supplied as args.

    Helper that wraps str.join() without having to pack strings in an iterable.

    "},{"location":"reference/cli/#jobbergate_cli.text_tools.copy_to_clipboard","title":"copy_to_clipboard","text":"
    copy_to_clipboard(text: str) -> bool\n

    Copy the provided text to the clipboard.

    If the clipboard is not available, return False. Otherwise, return True.

    "},{"location":"reference/cli/#jobbergate_cli.text_tools.dedent","title":"dedent","text":"
    dedent(text: str) -> str\n

    Dedents a paragraph after removing leading and trailing whitespace.

    "},{"location":"reference/cli/#jobbergate_cli.text_tools.dedent_all","title":"dedent_all","text":"
    dedent_all(*texts: str, join_str: str = '\\n') -> str\n

    Dedents each blob supplied as an argument and then joins them.

    "},{"location":"reference/cli/#jobbergate_cli.text_tools.indent","title":"indent","text":"
    indent(text: str, prefix: str = '    ', **kwargs) -> str\n

    Simple wrapper for the textwrap.indent() method but includes a default prefix.

    "},{"location":"reference/cli/#jobbergate_cli.text_tools.unwrap","title":"unwrap","text":"
    unwrap(text: str) -> str\n

    Unwraps a paragraph of text into a single line.

    The text may be indented.

    "},{"location":"reference/cli/#jobbergate_cli.time_loop","title":"time_loop","text":"

    Provide a time-loop class that can be used to to iterate during a given window of time.

    "},{"location":"reference/cli/#jobbergate_cli.time_loop.Tick","title":"Tick","text":"

    Bases: BaseModel

    A helper class describing a \"tick\".

    Contains a counter, elapsed time since the last tick, and total elapsed time.

    "},{"location":"reference/cli/#jobbergate_cli.time_loop.TimeLoop","title":"TimeLoop","text":"

    A special iterator that will iterate for a specified duration of time.

    Uses a progress meter to show the user how much time is left. Each iteration of the time-loop produces a tick.

    "},{"location":"reference/cli/#jobbergate_cli.time_loop.TimeLoop.__del__","title":"__del__","text":"
    __del__()\n

    Explicitly clear the progress meter if the time-loop is destroyed.

    "},{"location":"reference/cli/#jobbergate_cli.time_loop.TimeLoop.__init__","title":"__init__","text":"
    __init__(\n    duration: Union[pendulum.Duration, int],\n    message: str = \"Processing\",\n    color: str = \"green\",\n)\n

    Initialize the time-loop.

    Duration may be either a count of seconds or a pendulum.duration.

    "},{"location":"reference/cli/#jobbergate_cli.time_loop.TimeLoop.__iter__","title":"__iter__","text":"
    __iter__() -> TimeLoop\n

    Start the iterator.

    Creates and starts the progress meter

    "},{"location":"reference/cli/#jobbergate_cli.time_loop.TimeLoop.__next__","title":"__next__","text":"
    __next__() -> Tick\n

    Iterates the time loop and returns a tick.

    If the duration is complete, clear the progress meter and stop iteration.

    "},{"location":"reference/cli/#jobbergate_cli.time_loop.TimeLoop.clear","title":"clear","text":"
    clear()\n

    Clear the time-loop.

    Stops the progress meter (if it is set) and reset moments, counter, progress meter.

    "},{"location":"reference/core/","title":"Jobbergate Core Reference","text":""},{"location":"reference/core/#jobbergate_core","title":"jobbergate_core","text":"

    Jobbergate-core contains key components that are shared among sub-projects.

    "},{"location":"reference/core/#jobbergate_core.AuthenticationError","title":"AuthenticationError","text":"

    Bases: Buzz

    Base exception for errors related to authentication on Jobbergate.

    "},{"location":"reference/core/#jobbergate_core.JobbergateAuthHandler","title":"JobbergateAuthHandler dataclass","text":"

    High-level class used to manage authentication to requests to the Jobbergate-API

    After an instance of this class is created, it can be used to authenticate requests from both requests and httpx packages by passing it to the auth parameter on the request (see examples below).

    It just works out of the box. Behind the scenes, this procedure calls the :meth:JobbergateAuthHandler.acquire_access method to load the available tokens from the cache directory, it tries to refresh them if they are expired, or provides an URL to the user to login on the system.

    Notice all steps above are also available individually as public methods, allowing a fine control for advanced users.

    .. _requests: https://requests.readthedocs.io/en/latest/ .. _httpx: https://www.python-httpx.org/

    Parameters:

    Name Type Description Default cache_directory Path

    Directory to be used for the caching tokens.

    required login_domain str

    Domain used for the login.

    required login_audience str

    Audience of the login.

    required login_client_id str

    Client ID used for login.

    'default' Note

    These values depend on the identity provider used for authentication. Consult your system administrator or contact Omnivector support support@omnivector.solutions for further assistance.

    Note

    This class can interoperate with the tokens generated by the jobbergate-cli package, as long as they are stored in the same cache directory.

    Examples:

    The following example shows how to use the :meth:`JobbergateAuthHandler`\nclass to authenticate a request:\n\n>>> from pathlib import Path\n>>> import requests\n>>> from jobbergate_core import JobbergateAuthHandler\n>>> jobbergate_auth = JobbergateAuthHandler(\n...     cache_directory=Path(\".\"),\n...     login_domain=\"http://keycloak.local:8080/realms/jobbergate-local\",\n...     login_audience=\"https://local.omnivector.solutions\",\n...     login_client_id=\"cli\",\n... )\n>>> jobbergate_base_url = \"http://localhost:8000/jobbergate\"\n>>> response = requests.get(\n...     f\"{jobbergate_base_url}/applications\",\n...     auth=jobbergate_auth # this is the important part\n)\nLogin Here: http://keycloak.local:8080/realms/jobbergate-local/device?user_code=LMVJ-XOLG\n>>> response.raise_for_status()\n>>> print(f\"response = {response.json()}\")\n
    "},{"location":"reference/core/#jobbergate_core.JobbergateAuthHandler.__call__","title":"__call__","text":"
    __call__(request)\n

    This internal method allows the integration with the requests library.

    It is called automatically when the instance is passed to the auth parameter, see the examples in the class docstring.

    It adds the Authorization header to the request with the access token.

    "},{"location":"reference/core/#jobbergate_core.JobbergateAuthHandler.acquire_access","title":"acquire_access","text":"
    acquire_access() -> str\n

    High-level method to acquire a valid access token.

    This method will attempt, in order:

    • Use the internal access token from the instance
    • Load the tokens from the cache directory (see :meth:JobbergateAuthHandler.load_from_cache)
    • If the access token is unavailable or expired, refresh both tokens using the refresh token grant type (see :meth:JobbergateAuthHandler.refresh_tokens)
    • If the refresh token is unavailable or expired, login to generate both tokens (see :meth:JobbergateAuthHandler.login)

    Returns:

    Type Description str

    The bearer access token.

    Raises:

    Type Description AuthenticationError

    If all of the steps above fail to acquire a valid access token.

    "},{"location":"reference/core/#jobbergate_core.JobbergateAuthHandler.load_from_cache","title":"load_from_cache","text":"
    load_from_cache() -> None\n

    Load the tokens that are available at the cache directory.

    "},{"location":"reference/core/#jobbergate_core.JobbergateAuthHandler.login","title":"login","text":"
    login() -> None\n

    Login to Jobbergate.

    An URL will be printed to the console, the user must open it in a browser and provide their access credentials.

    After the login is completed, the tokens will be saved to the cache directory.

    "},{"location":"reference/core/#jobbergate_core.JobbergateAuthHandler.logout","title":"logout","text":"
    logout() -> None\n

    Logout from Jobbergate by clearing the loaded tokens and their cache on the disk.

    "},{"location":"reference/core/#jobbergate_core.JobbergateAuthHandler.refresh_tokens","title":"refresh_tokens","text":"
    refresh_tokens() -> None\n

    Refresh the tokens.

    After the refresh operation is completed, the tokens will be saved to the cache directory.

    Raises:

    Type Description AuthenticationError

    If the refresh token is missing or expired.

    "},{"location":"reference/core/#jobbergate_core.JobbergateAuthHandler.save_to_cache","title":"save_to_cache","text":"
    save_to_cache() -> None\n

    Save the tokens to the cache directory.

    Note

    This method will create the cache directory if it does not exist.

    "},{"location":"reference/core/#jobbergate_core.TokenError","title":"TokenError","text":"

    Bases: AuthenticationError

    Exception for errors related to tokens on Jobbergate.

    "},{"location":"reference/core/#jobbergate_core.auth","title":"auth","text":"

    Utilities for handling auth in Jobbergate.

    "},{"location":"reference/core/#jobbergate_core.auth.AuthenticationError","title":"AuthenticationError","text":"

    Bases: Buzz

    Base exception for errors related to authentication on Jobbergate.

    "},{"location":"reference/core/#jobbergate_core.auth.JobbergateAuthHandler","title":"JobbergateAuthHandler dataclass","text":"

    High-level class used to manage authentication to requests to the Jobbergate-API

    After an instance of this class is created, it can be used to authenticate requests from both requests and httpx packages by passing it to the auth parameter on the request (see examples below).

    It just works out of the box. Behind the scenes, this procedure calls the :meth:JobbergateAuthHandler.acquire_access method to load the available tokens from the cache directory, it tries to refresh them if they are expired, or provides an URL to the user to login on the system.

    Notice all steps above are also available individually as public methods, allowing a fine control for advanced users.

    .. _requests: https://requests.readthedocs.io/en/latest/ .. _httpx: https://www.python-httpx.org/

    Parameters:

    Name Type Description Default cache_directory Path

    Directory to be used for the caching tokens.

    required login_domain str

    Domain used for the login.

    required login_audience str

    Audience of the login.

    required login_client_id str

    Client ID used for login.

    'default' Note

    These values depend on the identity provider used for authentication. Consult your system administrator or contact Omnivector support support@omnivector.solutions for further assistance.

    Note

    This class can interoperate with the tokens generated by the jobbergate-cli package, as long as they are stored in the same cache directory.

    Examples:

    The following example shows how to use the :meth:`JobbergateAuthHandler`\nclass to authenticate a request:\n\n>>> from pathlib import Path\n>>> import requests\n>>> from jobbergate_core import JobbergateAuthHandler\n>>> jobbergate_auth = JobbergateAuthHandler(\n...     cache_directory=Path(\".\"),\n...     login_domain=\"http://keycloak.local:8080/realms/jobbergate-local\",\n...     login_audience=\"https://local.omnivector.solutions\",\n...     login_client_id=\"cli\",\n... )\n>>> jobbergate_base_url = \"http://localhost:8000/jobbergate\"\n>>> response = requests.get(\n...     f\"{jobbergate_base_url}/applications\",\n...     auth=jobbergate_auth # this is the important part\n)\nLogin Here: http://keycloak.local:8080/realms/jobbergate-local/device?user_code=LMVJ-XOLG\n>>> response.raise_for_status()\n>>> print(f\"response = {response.json()}\")\n
    "},{"location":"reference/core/#jobbergate_core.auth.JobbergateAuthHandler.__call__","title":"__call__","text":"
    __call__(request)\n

    This internal method allows the integration with the requests library.

    It is called automatically when the instance is passed to the auth parameter, see the examples in the class docstring.

    It adds the Authorization header to the request with the access token.

    "},{"location":"reference/core/#jobbergate_core.auth.JobbergateAuthHandler.acquire_access","title":"acquire_access","text":"
    acquire_access() -> str\n

    High-level method to acquire a valid access token.

    This method will attempt, in order:

    • Use the internal access token from the instance
    • Load the tokens from the cache directory (see :meth:JobbergateAuthHandler.load_from_cache)
    • If the access token is unavailable or expired, refresh both tokens using the refresh token grant type (see :meth:JobbergateAuthHandler.refresh_tokens)
    • If the refresh token is unavailable or expired, login to generate both tokens (see :meth:JobbergateAuthHandler.login)

    Returns:

    Type Description str

    The bearer access token.

    Raises:

    Type Description AuthenticationError

    If all of the steps above fail to acquire a valid access token.

    "},{"location":"reference/core/#jobbergate_core.auth.JobbergateAuthHandler.load_from_cache","title":"load_from_cache","text":"
    load_from_cache() -> None\n

    Load the tokens that are available at the cache directory.

    "},{"location":"reference/core/#jobbergate_core.auth.JobbergateAuthHandler.login","title":"login","text":"
    login() -> None\n

    Login to Jobbergate.

    An URL will be printed to the console, the user must open it in a browser and provide their access credentials.

    After the login is completed, the tokens will be saved to the cache directory.

    "},{"location":"reference/core/#jobbergate_core.auth.JobbergateAuthHandler.logout","title":"logout","text":"
    logout() -> None\n

    Logout from Jobbergate by clearing the loaded tokens and their cache on the disk.

    "},{"location":"reference/core/#jobbergate_core.auth.JobbergateAuthHandler.refresh_tokens","title":"refresh_tokens","text":"
    refresh_tokens() -> None\n

    Refresh the tokens.

    After the refresh operation is completed, the tokens will be saved to the cache directory.

    Raises:

    Type Description AuthenticationError

    If the refresh token is missing or expired.

    "},{"location":"reference/core/#jobbergate_core.auth.JobbergateAuthHandler.save_to_cache","title":"save_to_cache","text":"
    save_to_cache() -> None\n

    Save the tokens to the cache directory.

    Note

    This method will create the cache directory if it does not exist.

    "},{"location":"reference/core/#jobbergate_core.auth.Token","title":"Token dataclass","text":"

    Low-level class used to handling tokens.

    Parameters:

    Name Type Description Default cache_directory Path

    The directory used for cache.

    required label str

    The type of token.

    required content str

    The content of the token (default is \"\").

    ''

    Attributes:

    Name Type Description file_path Path

    The path to the file associated with the token. It is computed as <cache_directory>/<label>.token.

    data Dict[str, Any]

    Metadata decoded from the token's content are available in this dictionary. Expiration date and permissions are some examples of data that can be found.

    "},{"location":"reference/core/#jobbergate_core.auth.Token.bearer_token","title":"bearer_token property","text":"
    bearer_token: str\n

    Return the token with the Bearer prefix.

    "},{"location":"reference/core/#jobbergate_core.auth.Token.__post_init__","title":"__post_init__","text":"
    __post_init__()\n

    Post init method.

    "},{"location":"reference/core/#jobbergate_core.auth.Token.clear_cache","title":"clear_cache","text":"
    clear_cache() -> None\n

    Clear the token from cache by removing the file associated with it.

    "},{"location":"reference/core/#jobbergate_core.auth.Token.is_expired","title":"is_expired","text":"
    is_expired() -> bool\n

    Check if the token is expired.

    Returns:

    Type Description bool

    True if the token is expired, False otherwise.

    Raises:

    Type Description TokenError

    If the expiration date is not found.

    "},{"location":"reference/core/#jobbergate_core.auth.Token.is_valid","title":"is_valid","text":"
    is_valid() -> bool\n

    Verify if the token is valid, i.e., has content and is not expired.

    "},{"location":"reference/core/#jobbergate_core.auth.Token.load_from_cache","title":"load_from_cache","text":"
    load_from_cache() -> Token\n

    Load the token from the cache directory.

    Parameters:

    Name Type Description Default cache_directory

    The path to the cache directory.

    required label

    The type of token.

    required

    Returns:

    Type Description Token

    A new token with the content replaced.

    "},{"location":"reference/core/#jobbergate_core.auth.Token.replace","title":"replace","text":"
    replace(**changes) -> Token\n

    Create a new instance of the token with the changes applied.

    Other Parameters:

    Name Type Description content

    The content of the token.

    cache_directory

    The directory containing the cache.

    label

    The type of token.

    "},{"location":"reference/core/#jobbergate_core.auth.Token.save_to_cache","title":"save_to_cache","text":"
    save_to_cache() -> None\n

    Save the token to the cache file associated with it.

    Raises:

    Type Description TokenError

    If the parent directory does not exist.

    TokenError

    If there is an unknown error while saving the token.

    "},{"location":"reference/core/#jobbergate_core.auth.TokenError","title":"TokenError","text":"

    Bases: AuthenticationError

    Exception for errors related to tokens on Jobbergate.

    "},{"location":"reference/core/#jobbergate_core.auth.TokenType","title":"TokenType","text":"

    Bases: str, Enum

    The types of tokens available in the system are access and refresh.

    "},{"location":"reference/core/#jobbergate_core.auth.exceptions","title":"exceptions","text":""},{"location":"reference/core/#jobbergate_core.auth.exceptions.AuthenticationError","title":"AuthenticationError","text":"

    Bases: Buzz

    Base exception for errors related to authentication on Jobbergate.

    "},{"location":"reference/core/#jobbergate_core.auth.exceptions.TokenError","title":"TokenError","text":"

    Bases: AuthenticationError

    Exception for errors related to tokens on Jobbergate.

    "},{"location":"reference/core/#jobbergate_core.auth.handler","title":"handler","text":"

    Utilities for handling authentication in the Jobbergate system.

    "},{"location":"reference/core/#jobbergate_core.auth.handler.JobbergateAuthHandler","title":"JobbergateAuthHandler dataclass","text":"

    High-level class used to manage authentication to requests to the Jobbergate-API

    After an instance of this class is created, it can be used to authenticate requests from both requests and httpx packages by passing it to the auth parameter on the request (see examples below).

    It just works out of the box. Behind the scenes, this procedure calls the :meth:JobbergateAuthHandler.acquire_access method to load the available tokens from the cache directory, it tries to refresh them if they are expired, or provides an URL to the user to login on the system.

    Notice all steps above are also available individually as public methods, allowing a fine control for advanced users.

    .. _requests: https://requests.readthedocs.io/en/latest/ .. _httpx: https://www.python-httpx.org/

    Parameters:

    Name Type Description Default cache_directory Path

    Directory to be used for the caching tokens.

    required login_domain str

    Domain used for the login.

    required login_audience str

    Audience of the login.

    required login_client_id str

    Client ID used for login.

    'default' Note

    These values depend on the identity provider used for authentication. Consult your system administrator or contact Omnivector support support@omnivector.solutions for further assistance.

    Note

    This class can interoperate with the tokens generated by the jobbergate-cli package, as long as they are stored in the same cache directory.

    Examples:

    The following example shows how to use the :meth:`JobbergateAuthHandler`\nclass to authenticate a request:\n\n>>> from pathlib import Path\n>>> import requests\n>>> from jobbergate_core import JobbergateAuthHandler\n>>> jobbergate_auth = JobbergateAuthHandler(\n...     cache_directory=Path(\".\"),\n...     login_domain=\"http://keycloak.local:8080/realms/jobbergate-local\",\n...     login_audience=\"https://local.omnivector.solutions\",\n...     login_client_id=\"cli\",\n... )\n>>> jobbergate_base_url = \"http://localhost:8000/jobbergate\"\n>>> response = requests.get(\n...     f\"{jobbergate_base_url}/applications\",\n...     auth=jobbergate_auth # this is the important part\n)\nLogin Here: http://keycloak.local:8080/realms/jobbergate-local/device?user_code=LMVJ-XOLG\n>>> response.raise_for_status()\n>>> print(f\"response = {response.json()}\")\n
    __call__ 
    __call__(request)\n

    This internal method allows the integration with the requests library.

    It is called automatically when the instance is passed to the auth parameter, see the examples in the class docstring.

    It adds the Authorization header to the request with the access token.

    acquire_access 
    acquire_access() -> str\n

    High-level method to acquire a valid access token.

    This method will attempt, in order:

    • Use the internal access token from the instance
    • Load the tokens from the cache directory (see :meth:JobbergateAuthHandler.load_from_cache)
    • If the access token is unavailable or expired, refresh both tokens using the refresh token grant type (see :meth:JobbergateAuthHandler.refresh_tokens)
    • If the refresh token is unavailable or expired, login to generate both tokens (see :meth:JobbergateAuthHandler.login)

    Returns:

    Type Description str

    The bearer access token.

    Raises:

    Type Description AuthenticationError

    If all of the steps above fail to acquire a valid access token.

    load_from_cache 
    load_from_cache() -> None\n

    Load the tokens that are available at the cache directory.

    login 
    login() -> None\n

    Login to Jobbergate.

    An URL will be printed to the console, the user must open it in a browser and provide their access credentials.

    After the login is completed, the tokens will be saved to the cache directory.

    logout 
    logout() -> None\n

    Logout from Jobbergate by clearing the loaded tokens and their cache on the disk.

    refresh_tokens 
    refresh_tokens() -> None\n

    Refresh the tokens.

    After the refresh operation is completed, the tokens will be saved to the cache directory.

    Raises:

    Type Description AuthenticationError

    If the refresh token is missing or expired.

    save_to_cache 
    save_to_cache() -> None\n

    Save the tokens to the cache directory.

    Note

    This method will create the cache directory if it does not exist.

    "},{"location":"reference/core/#jobbergate_core.auth.token","title":"token","text":"

    Utilities for handling tokens on Jobbergate.

    "},{"location":"reference/core/#jobbergate_core.auth.token.Token","title":"Token dataclass","text":"

    Low-level class used to handling tokens.

    Parameters:

    Name Type Description Default cache_directory Path

    The directory used for cache.

    required label str

    The type of token.

    required content str

    The content of the token (default is \"\").

    ''

    Attributes:

    Name Type Description file_path Path

    The path to the file associated with the token. It is computed as <cache_directory>/<label>.token.

    data Dict[str, Any]

    Metadata decoded from the token's content are available in this dictionary. Expiration date and permissions are some examples of data that can be found.

    bearer_token property 
    bearer_token: str\n

    Return the token with the Bearer prefix.

    __post_init__ 
    __post_init__()\n

    Post init method.

    clear_cache 
    clear_cache() -> None\n

    Clear the token from cache by removing the file associated with it.

    is_expired 
    is_expired() -> bool\n

    Check if the token is expired.

    Returns:

    Type Description bool

    True if the token is expired, False otherwise.

    Raises:

    Type Description TokenError

    If the expiration date is not found.

    is_valid 
    is_valid() -> bool\n

    Verify if the token is valid, i.e., has content and is not expired.

    load_from_cache 
    load_from_cache() -> Token\n

    Load the token from the cache directory.

    Parameters:

    Name Type Description Default cache_directory

    The path to the cache directory.

    required label

    The type of token.

    required

    Returns:

    Type Description Token

    A new token with the content replaced.

    replace 
    replace(**changes) -> Token\n

    Create a new instance of the token with the changes applied.

    Other Parameters:

    Name Type Description content

    The content of the token.

    cache_directory

    The directory containing the cache.

    label

    The type of token.

    save_to_cache 
    save_to_cache() -> None\n

    Save the token to the cache file associated with it.

    Raises:

    Type Description TokenError

    If the parent directory does not exist.

    TokenError

    If there is an unknown error while saving the token.

    "},{"location":"reference/core/#jobbergate_core.auth.token.TokenType","title":"TokenType","text":"

    Bases: str, Enum

    The types of tokens available in the system are access and refresh.

    "},{"location":"reference/core/#jobbergate_core.version","title":"version","text":"

    Provide the version of the package.

    "}]} \ No newline at end of file +{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":"

    An Omnivector initiative

    "},{"location":"#jobbergate-documentation","title":"Jobbergate Documentation","text":"

    The following documentation provides a comprehensive overview of the Jobbergate system, detailing its purpose, installation process, and operational guidelines.

    Jobbergate serves as an advanced job templating and submission system, designed to seamlessly integrate with Slurm. This integration facilitates the efficient re-use and remote submission of job scripts to a Slurm cluster.

    At the heart of Jobbergate is its API, which acts as the pivotal control center for the entire system. This API interacts with an agent positioned alongside a Slurm cluster. This agent is responsible for establishing communication between both the Jobbergate API and the Slurm RESTful API. Furthermore, Jobbergate offers a Command Line Interface (CLI) to ensure users have an intuitive means of interacting with the system.

    Given that the API is cloud-based, users are granted the capability to modify jobs, dispatch them to affiliated clusters, and oversee their progress from any device with internet connectivity.

    Additionally, Jobbergate introduces a Python SDK named \"Jobbergate Core\". This SDK is equipped with tools tailored for automation and can be effortlessly integrated into any Python-based project.

    "},{"location":"authors/","title":"Authors","text":"

    Jobbergate is written and maintained by Omnivector, LLC. It is an open source project developed in collaboration with Scania, AB.

    "},{"location":"authors/#attribution","title":"Attribution","text":"

    This project began as a rewrite of the original Jobbergate project authored by Jimmy Hedman. The original provided the inspiration for an interactive tool used to gather template variable values to render jinja2 templates into Slurm job scripts.

    Building upon Jimmy's great idea, Jobbergate has grown into an entire system for managing and submitting reusable Slurm jobs, but the core idea of reusing templates combined with user input for the template values has remained the core of the project throughout its evolution.

    "},{"location":"authors/#jobbergate-development-team","title":"Jobbergate Development Team","text":"

    The Jobbergate project's main contributors are as follows:

    • Felipe Schuch \ud83d\udce7
    • James Beedy \ud83d\udce7
    • Lucas Carvalho
    • Matheus Tosta \ud83d\udce7
    • Tucker Beck \ud83d\udce7
    "},{"location":"authors/#get-in-touch","title":"Get in touch","text":"

    Contact Omnivector by email

    "},{"location":"tutorial/","title":"Tutorial","text":"

    Welcome to this step-by-step tutorial that introduces the basic functionalities of Jobbergate! to seamlessly upload a Job Script and submit it to a Slurm cluster using the Jobbergate CLI.

    In this walk-through, you will learn how to upload a Job Script and submit it to a Slurm cluster using the Jobbergate CLI. To accomplish this, we will guide you through the following steps:

    • Initiating a session by logging into the Jobbergate system
    • Uploading a basic Job Script to Jobbergate
    • Submitting the Job Script to the cluster
    • Reviewing the results and monitoring the status of your submitted job
    • Cleaning up by deleting the Job Script
    • Logging out of the Jobbergate system
    "},{"location":"tutorial/#getting-started","title":"Getting Started","text":"

    Before diving into the tutorial, there are some initial setup steps that are needed to ensure that your computer is prepared to run the tutorial locally. Make sure you have administrative access to your machine, as it's required for the setup process.

    "},{"location":"tutorial/#install-docker-compose","title":"Install docker-compose","text":"

    For this tutorial, we will be using an instance of Jobbergate that is deployed locally along-side a local Slurm cluster. We will set all this up using docker-compose. If you do not have it already, follow this guide to install docker-compose before you continue the tutorial.

    "},{"location":"tutorial/#update-the-hostfile","title":"Update the hostfile","text":"

    Next, you\u2019ll need to add the following line to your computer\u2019s hostfile:

    127.0.0.1 keycloak.local\n
    "},{"location":"tutorial/#for-linux-and-osx-users","title":"For Linux and OSX users","text":"
    • The hostfile is located at /etc/hosts.
    • Open a terminal.

    • Use this command to open the file in a text editor

      sudo nano /etc/hosts\n

      Note

      You may, of course, substitute nano by your editor of choice

    • Add the above line at the end of the file.

    • Save and close the file.
    "},{"location":"tutorial/#for-windows-users","title":"For Windows users","text":"
    • The hostfile can be found at c:\\windows\\system32\\drivers\\etc\\hosts.
    • Open Notepad as an administrator.
    • Open the hostfile in Notepad.
    • Append the above line to the file.
    • Save and exit.
    "},{"location":"tutorial/#clone-jobbergate-with-git","title":"Clone Jobbergate with git","text":"

    To run the Jobbergate and Slurm locally, you will first need a copy of the Jobbergate source code. The easiest way to get it is to use Git to download the source code repository from GitHub onto your machine.

    Git is a version control system that lets you manage and keep track of your source code history. If you haven't installed it yet, download and install Git using the instructions available here.

    With Git installed, you can now clone the Jobbergate source code from its GitHub repository. Cloning allows you to have a local copy (or clone) of the source code on your machine.

    Run the following command in your terminal:

    git clone git@github.com:omnivector-solutions/jobbergate.git\n

    Now you have a full copy of the Jobbergate source code including the Docker Compose configuration to stand up a local Slurm Cluster and the example Job Script we will be using for this tutorial.

    Next, switch to the directory in the source code that contains the Docker Compose configuration:

    cd jobbergate/jobbergate-composed\n
    "},{"location":"tutorial/#start-the-jobbergate-services","title":"Start the Jobbergate Services","text":"

    With the Jobbergate source code in place, it's time to initiate the Jobbergate Services and the local Slurm cluster using Docker Compose. Follow the steps outlined below to get things up and running.

    "},{"location":"tutorial/#start-up-the-services","title":"Start up the services","text":"

    Run the following command to build and start the services. The --build flag ensures that Docker Compose build the images before attempting to start the services. The --detach flag runs the services in the background so that you can run other commands in the terminal.

    docker-compose up --build --detach\n

    This operation might take a few minutes as it involves building the images and starting up all the associated services.

    "},{"location":"tutorial/#verify-the-status-of-the-services","title":"Verify the status of the services","text":"

    To confirm that all the services are running smoothly, execute the following command. It will list the status of all the services initiated by Docker Compose:

    docker-compose ps\n

    If the services are up and running as expected, you should see output similar to the following, indicating that all the services are in a healthy state

    NAME                                        COMMAND                  SERVICE               STATUS              PORTS\nc1                                          \"/usr/local/bin/slur\u2026\"   c1                    running             6818/tcp\nc2                                          \"/usr/local/bin/slur\u2026\"   c2                    running             6818/tcp\njobbergate-composed-cluster-agent-1         \"/agent/entrypoint.sh\"   cluster-agent         running\njobbergate-composed-db-1                    \"docker-entrypoint.s\u2026\"   db                    running             0.0.0.0:5432->5432/tcp\njobbergate-composed-jobbergate-api-1        \"/bin/sh -c /app/dev\u2026\"   jobbergate-api        running (healthy)   0.0.0.0:8000->80/tcp\njobbergate-composed-jobbergate-cli-1        \"python3\"                jobbergate-cli        exited (0)\njobbergate-composed-keycloak.local-1        \"/opt/keycloak/bin/k\u2026\"   keycloak.local        running             0.0.0.0:8080->8080/tcp, 8443/tcp\njobbergate-composed-minio-1                 \"/usr/bin/docker-ent\u2026\"   minio                 running             0.0.0.0:9000-9001->9000-9001/tcp\njobbergate-composed-minio-create-bucket-1   \"/create-bucket.sh\"      minio-create-bucket   exited (1)\nmysql                                       \"docker-entrypoint.s\u2026\"   mysql                 running             3306/tcp, 33060/tcp\nslurmctld                                   \"/usr/local/bin/slur\u2026\"   slurmctld             running             6817/tcp\nslurmdbd                                    \"/usr/local/bin/slur\u2026\"   slurmdbd              running             6819/tcp\nslurmrestd                                  \"/usr/local/bin/slur\u2026\"   slurmrestd            running             0.0.0.0:6820->6820/tcp\n

    The STATUS for each service should be \"running\" except for the minio-create-bucket and jobbergate-cli services that should be \"exited\".

    "},{"location":"tutorial/#confirm-jobbergate-cli-availability","title":"Confirm Jobbergate CLI availability","text":"

    Since this tutorial relies on running commands in the Jobbergate CLI, it's essential to verify that the CLI is available and working as expected at this juncture.

    First, initiate a connection to the jobbergate-cli container by executing the following command. This gives you direct access to the CLI.

    docker-compose run jobbergate-cli bash\n

    Upon successful connection, your command prompt should change to reflect that you're inside the container. It will look something like this:

    root@e226a9a401d1:/app#\n

    This confirms that you're now operating within the jobbergate-cli container environment.

    Next, we need to make sure that the Jobbergate CLI is available and accepting commands. Test this by listing the available commands in Jobbergate CLI with the --help option:

    jobbergate --help\n

    The command above will yield a detailed description of the CLI's usage and the variety of sub-commands it provides:

     Usage: jobbergate [OPTIONS] COMMAND [ARGS]...\n\n Welcome to the Jobbergate CLI!\n More information can be shown for each command listed below by running it with the --help option.\n\n\u256d\u2500 Options \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 --verbose               --no-verbose                                     Enable verbose logging to the terminal      \u2502\n\u2502                                                                          [default: no-verbose]                       \u2502\n\u2502 --full                  --no-full                                        Print all fields from CRUD commands         \u2502\n\u2502                                                                          [default: no-full]                          \u2502\n\u2502 --raw                   --no-raw                                         Print output from CRUD commands as raw json \u2502\n\u2502                                                                          [default: no-raw]                           \u2502\n\u2502 --version               --no-version                                     Print the version of jobbergate-cli and     \u2502\n\u2502                                                                          exit                                        \u2502\n\u2502                                                                          [default: no-version]                       \u2502\n\u2502 --install-completion                    [bash|zsh|fish|powershell|pwsh]  Install completion for the specified shell. \u2502\n\u2502                                                                          [default: None]                             \u2502\n\u2502 --show-completion                       [bash|zsh|fish|powershell|pwsh]  Show completion for the specified shell, to \u2502\n\u2502                                                                          copy it or customize the installation.      \u2502\n\u2502                                                                          [default: None]                             \u2502\n\u2502 --help                                                                   Show this message and exit.                 \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n\u256d\u2500 Commands \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 applications        Commands to interact with applications                                                           \u2502\n\u2502 job-scripts         Commands to interact with job scripts                                                            \u2502\n\u2502 job-submissions     Commands to interact with job submissions                                                        \u2502\n\u2502 login               Log in to the jobbergate-cli by storing the supplied token argument in the cache.                \u2502\n\u2502 logout              Logs out of the jobbergate-cli. Clears the saved user credentials.                               \u2502\n\u2502 show-token          Show the token for the logged in user.                                                           \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n
    "},{"location":"tutorial/#log-in-to-jobbergate","title":"Log in to Jobbergate","text":"

    To begin working with Jobbergate data, you must first sign into the system. For the purpose of this tutorial, there's just one user available. We'll solely focus on this user in this guide, but should you wish to add more users, you can do so by accessing the Keycloak server (details provided in the Appendix).

    To log in using the Jobbergate CLI, execute the following command:

    jobbergate login\n

    The CLI will provide a URL for you to log into your account:

    \u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Waiting for login \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502                                                                                                                   \u2502\n\u2502   To complete login, please open the following link in a browser:                                                 \u2502\n\u2502                                                                                                                   \u2502\n\u2502     http://keycloak.local:8080/realms/jobbergate-local/device?user_code=CZAU-TZAH                                 \u2502\n\u2502                                                                                                                   \u2502\n\u2502   Waiting up to 5.0 minutes for you to complete the process...                                                    \u2502\n\u2502                                                                                                                   \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n\nWaiting for web login... \u2501\u257a\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501   3% 0:04:50\n

    Open the URL shown in a browser and log in as \"local-user\":

    • username: \"local-user\"
    • password: \"local\"

    When prompted, grant all the requested access privileges to the CLI. Once you have finished, the CLI will show that you have successfully logged in:

    \u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Logged in! \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502                                                                                                                   \u2502\n\u2502   User was logged in with email 'local-user@jobbergate.local'                                                     \u2502\n\u2502                                                                                                                   \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n

    You are now logged in through the CLI! Your auth token will be cached automatically for you, so you should not need to log in again for some time. However, be aware that your session does expire; you will have to log in again to get a new token. If this happens, the CLI will alert you that your token is invalid. When you receive this notification, you will need to log in anew.

    "},{"location":"tutorial/#upload-a-job-script-to-jobbergate","title":"Upload a Job Script to Jobbergate","text":"

    Job Scripts are integral to Jobbergate, serving as the foundation for running simulations on our cluster. To initiate a simulation, your first task is to upload the Job Script to the Jobbergate API.

    Within each Job Script, an entrypoint file is designated. This is the specific script that Slurm executes to commence the simulation on the cluster.

    "},{"location":"tutorial/#get-the-example-script","title":"Get the example script","text":"

    To keep this tutorial focused on using Jobbergate and not any of the complexities of simulations or operating a cluster, we will use a very basic example job script. We will need a copy of this script where the jobbergate-cli can access it. Since it's a small script, we can just copy/paste it into the container where we are accessing the jobbergate-cli.

    In the terminal where you were typing jobbergate commands, enter this command:

    cat > simple-job-script.py\n

    Paste the contents of the job script and then press ctrl-d on your keyboard. This will create a saved copy of the job script that's ready to submit with the jobbergate-cli. To ensure that the command sequence captured the intended script contents, execute the following command to review the job script:

    cat simple-job-script.python3\n

    The script should appear exactly as you see it on the link above.

    "},{"location":"tutorial/#create-the-job-script-from-the-example","title":"Create the Job Script from the example","text":"

    Now it's time to create a Job Script entry within the Jobbergate system. We'll use the create subcommand associated with the job-scripts command. To view all the options that come with this sub-command, you can use the --help option:

    jobbergate job-scripts create --help\n

    Now, let's create the Job Script. In your terminal, type:

    jobbergate job-scripts create --name=tutorial --job-script-path=simple-job-script.py\n

    You should see output like this indicating that the Job Script was successfully created:

                     Created Job Script\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key            \u2503 Value                            \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 id             \u2502 1                                \u2502\n\u2502 application_id \u2502 None                             \u2502\n\u2502 name           \u2502 tutorial                         \u2502\n\u2502 description    \u2502                                  \u2502\n\u2502 owner_email    \u2502 local-user@jobbergate.local-mail \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n

    Great, your Job Script is now prepared and ready for submission to the cluster!

    Note

    Keep track of the id value produced by your command. The tutorial text assumes that it is \"1\", but it may be different if you have done the tutorial before or had to restart!

    To confirm that the Job Script has been uploaded correctly, you can review the file content using the show-files subcommand:

    jobbergate job-scripts show-files --id=1\n

    The file should appear exactly as it does on the link above.

    "},{"location":"tutorial/#submit-a-job-script-to-the-cluster","title":"Submit a Job Script to the cluster","text":"

    With the Job Script ready, the next step is to submit it to the Slurm cluster. In this tutorial, a cluster named local-slurm is already attached and available for use. We will specify this cluster name when submitting the Job Script to ensure it is executed on the appropriate cluster.

    "},{"location":"tutorial/#create-the-job-submission","title":"Create the Job Submission","text":"

    We will use the create subcommand of the job-submissions command to submit the job to the cluster. To see all the options available for this command, we can use the --help option again:

    jobbergate job-submissions create --help\n

    For the tutorial, we need to issue the following command:

    jobbergate job-submissions create --name=tutorial --job-script-id=1 --cluster-name=local-slurm\n

    The command should produce output that looks like this:

                       Created Job Submission\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key                        \u2503 Value                       \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 id                         \u2502 1                           \u2502\n\u2502 job_script_id              \u2502 1                           \u2502\n\u2502 client_id                  \u2502 local-slurm                 \u2502\n\u2502 slurm_job_id               \u2502 None                        \u2502\n\u2502 execution_directory        \u2502 None                        \u2502\n\u2502 job_submission_name        \u2502 tutorial                    \u2502\n\u2502 job_submission_description \u2502 None                        \u2502\n\u2502 job_submission_owner_email \u2502 local-user@jobbergate.local \u2502\n\u2502 status                     \u2502 CREATED                     \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n

    Info

    The Job Submission was successfully created! However, it has not submitted to the cluster yet, and thus slurm_job_id is still None. This will happen when the Jobbergate Agent that is running remotely in the cluster pulls all \"CREATED\" Job Submissions down from the API and submits them to Slurm one by one.

    Note

    Again, be careful to use the correct id produced by this command for the remainder of the tutorial!

    "},{"location":"tutorial/#check-the-status-of-the-submitted-job","title":"Check the status of the submitted job","text":"

    We can look up the status of a Job Submission using the following command:

    jobbergate job-submissions get-one --id=1\n

    This command should produce output that looks like:

                           Job Submission\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key                        \u2503 Value                       \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 id                         \u2502 1                           \u2502\n\u2502 job_script_id              \u2502 1                           \u2502\n\u2502 client_id                  \u2502 local-slurm                 \u2502\n\u2502 slurm_job_id               \u2502 1                           \u2502\n\u2502 execution_directory        \u2502 None                        \u2502\n\u2502 job_submission_name        \u2502 tutorial                    \u2502\n\u2502 job_submission_description \u2502 None                        \u2502\n\u2502 job_submission_owner_email \u2502 local-user@jobbergate.local \u2502\n\u2502 status                     \u2502 SUBMITTED                   \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n

    If the status reported by your command is CREATED, don't worry! The Jobbergate Agent just hasn't retrieved and submitted the job script yet. Wait a few more seconds and try again. You should now see the status change to SUBMITTED.

    When the Job Submission status shifts to SUBMITTED, it indicates that the Jobbergate Agent has retrieved the Job Script and submitted it to the local-slurm cluster. This status will persist until the completion of the Job Script's execution. The Jobbergate Agent continuously monitors the job's progress within slurm, and, upon its completion, will update the Job Submission status to COMPLETE.

    "},{"location":"tutorial/#check-the-results-of-the-job","title":"Check the results of the job","text":"

    In this tutorial, we have locally mounted a \"fake\" NFS folder to contain the output from the job running in slurm. When the job finishes running, it will produce an output file in this folder. First we need to verify that the file was produced by listing the contents of the nfs directory:

    ls /nfs\n

    If the job completed, you should see a file in the /nfs directory named simple-output.txt. Check the contents of the file with a simple cat command:

    cat /nfs/simple-output.txt\n

    It should look look like:

    Simple output from c1\n

    It's possible that the output says it came from c2 if slurm ran the job on the c2 compute node instead of c1.

    "},{"location":"tutorial/#delete-the-resources","title":"Delete the resources","text":"

    Sometimes it is useful to remove resources that have been created in Jobbergate.

    For instance, start by deleting the Job Submission:

    $ jobbergate job-submissions delete --id=1\n\n\u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Job submission delete succeeded \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502                                                                                                                   \u2502\n\u2502   The job submission was successfully deleted.                                                                    \u2502\n\u2502                                                                                                                   \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n

    Then delete the Job Script:

    $ jobbergate job-scripts delete --id=1\n\n\u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Job script delete succeeded \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502                                                                                                                   \u2502\n\u2502   The job script was successfully deleted.                                                                        \u2502\n\u2502                                                                                                                   \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n
    "},{"location":"tutorial/#log-out-of-the-jobbergate-system","title":"Log out of the Jobbergate system","text":"

    You have completed the tutorial. Try logging out of Jobbergate now:

    $ jobbergate logout\n\n\u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Logged out \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502                                                                                                                   \u2502\n\u2502   User was logged out.                                                                                            \u2502\n\u2502                                                                                                                   \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n

    This will clear any cached tokens, and any subsequent Jobbergate commands will require you to log in again.

    "},{"location":"tutorial/#appendix","title":"Appendix","text":""},{"location":"tutorial/#keycloak-ui","title":"Keycloak UI","text":"

    You can connect to the Keycloak UI to create additional realms, clients, and users. However, the use of Keycloak is a rather large topic that goes outside the scope of this Tutorial.

    To get started, you can connect to the Keycloak UI through a browser if the server is running as a part of the docker-compose cluster using this local URL. To log in as administrator use these credentials:

    • username: admin
    • password: admin
    "},{"location":"developer_guide/ci/","title":"Continuous Integration","text":"

    Jobbergate employs GitHub actions for its continuous integration processes. Detailed descriptions of these actions are provided on this page.

    "},{"location":"developer_guide/ci/#automated-quality-assurance","title":"Automated Quality Assurance","text":"

    Jobbergate's git repository incorporates a GitHub Action, specified in test_on_push.yaml, which is designed to execute our quality assurance tools across all Jobbergate sub-projects simultaneously. The action is activated anytime a new commit is pushed to the main branch or whenever a pull request is submitted.

    The suite of quality assurance tools encompasses unit tests, code coverage, linters, code formatters, and static type checkers. Comprehensive documentation about each tool is available in the Quality Assurance Tools section.

    "},{"location":"developer_guide/ci/#automated-publication-to-pypi","title":"Automated Publication to PyPI","text":"

    The major components of Jobbergate are published on PyPI, the Python Package Index. They are available at:

    • jobbergate-api
    • jobbergate-cli
    • jobbergate-agent
    • jobbergate-core

    These packages are automatically published to PyPI by three linked GitHub Actions that are detailed below.

    "},{"location":"developer_guide/ci/#prepare-for-release","title":"Prepare for release","text":"

    The first action involved in publication is the prepare_release.yaml) action. It is triggered manually on github through a \"workflow dispatch event\" whenever new features or fixes need to be published.

    The action takes two arguments that must be supplied by the user. They are:

    • Use workflow from: The branch from which the release will be created. The default is main, and it's highly recommended that releases are cut from this branch in order to keep a linear commit history between releases and pre-releases.
    • Release Type: This will describe the release type that will be created. Because Jobbergate uses semantic versioning, it's important to carefully select the correct type of release. For mor information on release types, please see the Poetry documentation to learn more.

    Once activated, this action:

    • Uses Poetry to bump the version number of all the Jobbergate sub-packages according to the release type selected.
    • Checks if the new version number is synchronized between the sub-packages, and fails if they are not.
    • Creates a new dated entry for the new release on each of the sub-packages' changelog files from the contents of the \"Unreleased\" section.
    • Creates a new branch named prepare-release/<version>.
    • Opens a draft pull request titled Release <version>.

    In this way, all the changes above can be reviewed before the release is published, and all quality assurance tests are executed for the pull request.

    The remaining steps of the workflow are chained automatically once the PR is accepted and merged into main.

    "},{"location":"developer_guide/ci/#create-a-new-tag","title":"Create a new tag","text":"

    The next action in the sequence is the tag_on_merged_pull_request.yaml action. Once the automatically created release PR is merged into the main branch, this action is triggered. It creates and pushes a new git tag to GitHub. The tag is based on the new version number for the release.

    "},{"location":"developer_guide/ci/#publish-on-tag","title":"Publish on Tag","text":"

    The final action is publish_on_tag.yaml This action is triggered when a new version tag is pushed to the repository. It first double checks if the tag matches the version number of each Jobbergate component, and then it builds and publishes the packages on PyPI.

    "},{"location":"developer_guide/dev_tools/","title":"API Dev Tools","text":"

    The Jobbergate API sub-project is equipped with a few tools designed to assist with some everyday development tasks. These can help streamline the process of setting up and interacting with the API.

    The dev-tools are shipped as a CLI program that can be invoked via Poetry within the project. All of the commands will operate within the virtual environment set up by Poetry.

    "},{"location":"developer_guide/dev_tools/#invoking-dev-tools","title":"Invoking dev-tools","text":"

    To invoke the dev tools, you must execute the commands from the home directory for the jobbergate-api. To see some information about the dev-tools, execute:

    poetry run dev-tools --help\n

    This will provide some help output that shows what options and sub-commands are available:

    Usage: dev-tools [OPTIONS] COMMAND [ARGS]...\n\nOptions:\n  --install-completion [bash|zsh|fish|powershell|pwsh]\n                                  Install completion for the specified shell.\n  --show-completion [bash|zsh|fish|powershell|pwsh]\n                                  Show completion for the specified shell, to\n                                  copy it or customize the installation.\n  --help                          Show this message and exit.\n\nCommands:\n  db\n  dev-server  Start a development server locally.\n  show-env    Print out the current environment settings.\n

    The --help option is available for all of the subcommands provided in dev-tools.

    "},{"location":"developer_guide/dev_tools/#the-db-subcommand","title":"The db subcommand","text":"

    There are a few convenience methods in the dev-tools for interacting with Jobbergate API's PostgreSQL database. These tools are found in the db subcommand. To see more info about this sub-command, run:

    poetry run dev-tools db --help\n
    "},{"location":"developer_guide/dev_tools/#the-login-subcommand","title":"The login subcommand","text":"

    This command allows you to log in to the database that your Jobbergate API is configured to connect with. It allows you to login to databases, regardless of whether they are locally hosted via Docker or situated on a remote PostgreSQL server. this ensures seamless access to any database that the Jobbergate API is configured to connect with.

    To log in to the database, execute this command:

    poetry run dev-tools db login\n

    The command will show some debug output including the URL of the database to which it is connecting and will then show a REPL connection to the database:

    2022-09-07 15:52:02.089 | DEBUG    | dev_tools.db:login:26 - Logging into database: postgresql://compose-db-user:compose-db-pswd@localhost:5432/compose-db-name\nServer: PostgreSQL 14.1 (Debian 14.1-1.pgdg110+1)\nVersion: 3.4.1\nHome: http://pgcli.com\ncompose-db-name>\n
    "},{"location":"developer_guide/dev_tools/#the-migrate-subcommand","title":"The migrate subcommand","text":"

    This command uses alembic to generate a migration script to bring the current database (described by the environment) up to date with the SQLAlchemy models specified in the Jobbergate API source code.

    To invoke the migration script generation, execute:

    poetry run dev-tools db migrate --message=\"An example migration\"\n

    Some logging info will be produced, including the location of the new migration script:

    2022-09-07 15:58:09.725 | DEBUG    | dev_tools.db:migrate:79 - Creating migration with message: An example migration\nINFO  [alembic.runtime.migration] Context impl PostgresqlImpl.\nINFO  [alembic.runtime.migration] Will assume transactional DDL.\nINFO  [alembic.ddl.postgresql] Detected sequence named 'applications_id_seq' as owned by integer column 'applications(id)', assuming SERIAL and omitting\nINFO  [alembic.ddl.postgresql] Detected sequence named 'job_scripts_id_seq' as owned by integer column 'job_scripts(id)', assuming SERIAL and omitting\nINFO  [alembic.ddl.postgresql] Detected sequence named 'job_submissions_id_seq' as owned by integer column 'job_submissions(id)', assuming SERIAL and omitting\n  Generating /home/dusktreader/git-repos/omnivector/jobbergate/jobbergate-api/alembic/versions/20220907_155809--c275de463a90_an_example_migration.py ...  done\n  Running post write hook \"black\" ...\nreformatted /home/dusktreader/git-repos/omnivector/jobbergate/jobbergate-api/alembic/versions/20220907_155809--c275de463a90_an_example_migration.py\n\nAll done! \u2728 \ud83c\udf70 \u2728\n1 file reformatted.\n  done\n  Running post write hook \"isort\" ...\nFixing /home/dusktreader/git-repos/omnivector/jobbergate/jobbergate-api/alembic/versions/20220907_155809--c275de463a90_an_example_migration.py\n  done\n

    The generated migration should always be reviewed before it is committed to the repository.

    It is also possible to produce a blank migration if you need to execute some raw SQL or write an Alembic script by hand. Just pass the --blank parameter on the command line:

    poetry run dev-tools db migrate --blank --message=\"A blank migration\"\n
    "},{"location":"developer_guide/dev_tools/#the-upgrade-subcommand","title":"The upgrade subcommand","text":"

    This subcommand is used to apply a database migration to the database that the Jobbergate API is configured to connect with.

    By default, it will apply all the migrations that have not yet been applied to the database.

    To apply the migrations, execute the command:

    poetry run dev-tools db upgrade\n

    It will produce some logging output that shows what migrations were applied:

    2022-09-07 16:05:46.315 | DEBUG    | dev_tools.db:upgrade:89 - Upgrading database...\nINFO  [alembic.runtime.migration] Context impl PostgresqlImpl.\nINFO  [alembic.runtime.migration] Will assume transactional DDL.\nINFO  [alembic.runtime.migration] Running upgrade d22da0741b7f -> c275de463a90, An example migration\n

    If you wish to only upgrade the database to a specific migration, you can pass that migration's id to the --target param.

    "},{"location":"developer_guide/dev_tools/#the-show-env-subcommand","title":"The show-env subcommand","text":"

    This command will show how the Jobbergate API is configured through its environment settings. To see the environment, execute this command:

    poetry run dev-tools show-env\n

    The output that the command produces will look something like:

    Jobbergate settings:\n  DEPLOY_ENV: LOCAL\n  LOG_LEVEL: DEBUG\n  DATABASE_HOST: localhost\n  DATABASE_USER: compose-db-user\n  DATABASE_PSWD: compose-db-pswd\n  DATABASE_NAME: compose-db-name\n  DATABASE_PORT: 5432\n  TEST_DATABASE_HOST: localhost\n  TEST_DATABASE_USER: test-user\n  TEST_DATABASE_PSWD: test-pswd\n  TEST_DATABASE_NAME: test-db\n  TEST_DATABASE_PORT: 5433\n  S3_BUCKET_NAME: jobbergate-k8s-staging\n  S3_ENDPOINT_URL: None\n  ARMASEC_DOMAIN: localhost:9080/realms/master/protocol/openid-connect\n  ARMASEC_USE_HTTPS: True\n  ARMASEC_AUDIENCE: https://local.omnivector.solutions\n  ARMASEC_DEBUG: True\n  ARMASEC_ADMIN_DOMAIN: None\n  ARMASEC_ADMIN_AUDIENCE: None\n  ARMASEC_ADMIN_MATCH_KEY: None\n  ARMASEC_ADMIN_MATCH_VALUE: None\n  IDENTITY_CLAIMS_KEY: https://omnivector.solutions\n  SENTRY_DSN: None\n  SENTRY_SAMPLE_RATE: 1.0\n  MAX_UPLOAD_FILE_SIZE: 104857600\n  SENDGRID_FROM_EMAIL: None\n  SENDGRID_API_KEY: None\n

    The command can also produce the output as JSON if needed by passing the --json flag:

    poetry run dev-tools show-env --json\n

    The JSON output will look something like:

    {\"DEPLOY_ENV\": \"LOCAL\", \"LOG_LEVEL\": \"DEBUG\", \"DATABASE_HOST\": \"localhost\", \"DATABASE_USER\": \"compose-db-user\", \"DATABASE_PSWD\": \"compose-db-pswd\", \"DATABASE_NAME\": \"compose-db-name\", \"DATABASE_PORT\": 5432, \"TEST_DATABASE_HOST\": \"localhost\", \"TEST_DATABASE_USER\": \"test-user\", \"TEST_DATABASE_PSWD\": \"test-pswd\", \"TEST_DATABASE_NAME\": \"test-db\", \"TEST_DATABASE_PORT\": 5433, \"S3_BUCKET_NAME\": \"jobbergate-k8s-staging\", \"S3_ENDPOINT_URL\": null, \"ARMASEC_DOMAIN\": \"localhost:9080/realms/master/protocol/openid-connect\", \"ARMASEC_USE_HTTPS\": true, \"ARMASEC_AUDIENCE\": \"https://local.omnivector.solutions\", \"ARMASEC_DEBUG\": true, \"ARMASEC_ADMIN_DOMAIN\": null, \"ARMASEC_ADMIN_AUDIENCE\": null, \"ARMASEC_ADMIN_MATCH_KEY\": null, \"ARMASEC_ADMIN_MATCH_VALUE\": null, \"IDENTITY_CLAIMS_KEY\": \"https://omnivector.solutions\", \"SENTRY_DSN\": null, \"SENTRY_SAMPLE_RATE\": 1.0, \"MAX_UPLOAD_FILE_SIZE\": 104857600, \"SENDGRID_FROM_EMAIL\": null, \"SENDGRID_API_KEY\": null}\n
    "},{"location":"developer_guide/dev_tools/#the-dev-server-subcommand","title":"The dev-server subcommand","text":"

    This command starts up a local development server for the Jobbergate API. It will be created using the configuration set up in the environment settings. This command is especially useful if you want to run the API locally but connect to remote services such as a database and s3 hosted on AWS.

    To start the server, run:

    poetry run dev-tools dev-server\n

    The command will produce some logging output that looks like this:

    2022-09-07 16:15:05.830 | INFO     | dev_tools.dev_server:dev_server:50 - Waiting for the database\n2022-09-07 16:15:05.830 | DEBUG    | dev_tools.dev_server:_wait_for_db:23 - database url is: postgresql://compose-db-user:compose-db-pswd@localhost:5432/compose-db-name\n2022-09-07 16:15:05.830 | DEBUG    | dev_tools.dev_server:_wait_for_db:26 - Checking health of database at postgresql://compose-db-user:compose-db-pswd@localhost:5432/compose-db-name: Attempt #0\nINFO:     Will watch for changes in these directories: ['/home/dusktreader/git-repos/omnivector/jobbergate/jobbergate-api']\nINFO:     Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)\nINFO:     Started reloader process [27314] using statreload\n2022-09-07 16:15:06.555 | INFO     | jobbergate_api.main:<module>:39 - Skipping Sentry\nINFO:     Started server process [27319]\nINFO:     Waiting for application startup.\n2022-09-07 16:15:06.587 | INFO     | jobbergate_api.main:init_logger:71 - Logging configured \ud83d\udcdd Level: DEBUG\n2022-09-07 16:15:06.587 | DEBUG    | jobbergate_api.main:init_database:79 - Initializing database\nINFO:     Application startup complete.\n

    There are additional options that can control some of the details of the settings of the dev server. These can be examined with the --help flag:

    poetry run dev-tools dev-server --help\n

    The dev server options will be printed like:

    Usage: dev-tools dev-server [OPTIONS]\n\n  Start a development server locally.\n\nOptions:\n  --db-wait-count INTEGER   How many times to attempt a check  [default: 3]\n  --db-wait-interval FLOAT  Seconds to wait between checks  [default: 5.0]\n  --port INTEGER            The port where the server should listen  [default:\n                            5000]\n  --log-level TEXT          The level to log uvicorn output  [default: DEBUG]\n  --help                    Show this message and exit.\n

    The --db-wait-* flags are used to make the dev server wait for the dev database to become available. These are mostly useful in the context of docker-compose.

    It should also be noted that a development uvicorn server will automatically reload the app if the source files of the app change. This is very helpful for debugging behavior in the app without having to manually stop and start the app after every source code modification.

    "},{"location":"developer_guide/integration_testing/","title":"Integration Testing","text":"

    While conducting integration testing for Jobbergate, it's critical to examine the entire cycle of the platform, ranging from the creation of an Application to remote Job Submission via the Jobbergate Agent.

    To test most of the platforms functionality, the docker-compose setup located in the Jobbergate Composed sub-project is sufficient. Begin by referring to the guide in that sub-project's README. Pay close attention to the execution of Jobbergate CLI commands as they play a significant role in integration testing. For testing you can use the pre-configured user credentials:

    • Username: local-user
    • Password: local

    Integration testing should cover the following work-flows:

    • Logging in through the CLI
    • Creating an Application
    • Querying a single Application
    • Updating an Application
    • Rendering a Job Script from an Application
    • Updating a Job Script
    • Submitting a Job Script
    • Verifying a Job Submission
    • Deleting the Job Submission, Job Script, and Application
    • Logging out through the CLI
    "},{"location":"developer_guide/integration_testing/#setup","title":"Setup","text":"

    To begin, you will need two separate terminals open. Change directory to the jobbergate-composed sub-project of the top-level jobbergate folder.

    First, you need to start up the Jobbergate platform with docker-compose. In one of your terminals, run the following command:

    docker-compose up --build\n

    Once all the services are started, jump into the prepared jobbergate-cli container to execute CLI commands. To do so, execute this command in the other terminal you have prepared:

    docker-compose run jobbergate-cli bash\n

    Now you may start executing commands with the Jobbergate CLI.

    To assist with some of the commands below, create a NAME environment variable that will help to identify resources that you create during the process. You should set the value based on the current date so that the associated resources are easy to identify. Run the following command to set it:

    export NAME=\"test--$(whoami)--$(date -I)\"\n

    You have now created a test name like test--tbeck--2023-10-13.

    "},{"location":"developer_guide/integration_testing/#logging-in-through-the-cli","title":"Logging in through the CLI","text":"

    The first work-flow you will test covers the auth mechanics of both the CLI and the API.

    Run the following command in the Jobbergate CLI:

    jobbergate login\n

    Next, open the link that is printed out and log in as local-user (password \"local\"). If asked, grant all of the permissions.

    Verify that the CLI reports that the user has been successfully logged in.

    At this point, verify that the token that has been retrieved for the user is correct.

    Run the following command in the CLI:

    jobbergate show-token --decode\n

    This command will pretty print the payload of the token. Verify that it contains:

    • \"view\" and \"edit\" permissions for job-templates, job-scripts, and job-submissions
    • email equalling \"local-user@jobberate.local\"
    • aud includes \"https://local.omnivector.solutions\"
    • azp equals \"jobbergate-cli\"
    "},{"location":"developer_guide/integration_testing/#creating-an-application","title":"Creating an Application","text":"

    Next, test the command to create an Application through the CLI, and verify that the resource is created in the database. Also, verify that the files are successfully uploaded to the file store.

    For integration testing, use the built-in simple application. example. This example application has 3 simple template variables, and, when submitted, the rendered Job Script simply prints the values of those variables.

    Run the following command in the Jobbergate CLI:

    jobbergate applications create --name=$NAME --identifier=$NAME --application-path=/example\n

    Verify that output shows that a single application was inserted and that the files were uploaded:

                        Created Application\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key                  \u2503 Value                            \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 id                   \u2502 1                                \u2502\n\u2502 name                 \u2502 test--root--2023-10-13           \u2502\n\u2502 owner_email          \u2502 local-user@jobbergate.local-mail \u2502\n\u2502 is_archived          \u2502 False                            \u2502\n\u2502 description          \u2502                                  \u2502\n\u2502 identifier           \u2502 test--root--2023-10-13           \u2502\n\u2502 application_uploaded \u2502 True                             \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n
    "},{"location":"developer_guide/integration_testing/#querying-a-single-application","title":"Querying a single Application","text":"

    Next, verify that we can look up a single Application by both its id and its identifier. Also include the --full argument to the base jobbergate command so that the output will show all the fields in the database including the source file, the config, and the timestamps.

    First, fetch the Application by id using the following command in the CLI:

    jobbergate --full applications get-one --id=1\n

    The output should look something like this:

    \u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key            \u2503 Value                                                                                                                                 \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 id             \u2502 1                                                                                                                                     \u2502\n\u2502 name           \u2502 test--root--2023-10-13                                                                                                                \u2502\n\u2502 owner_email    \u2502 local-user@jobbergate.local-mail                                                                                                      \u2502\n\u2502 created_at     \u2502 2023-10-13T19:44:21.935886                                                                                                            \u2502\n\u2502 updated_at     \u2502 2023-10-13T19:44:21.958325                                                                                                            \u2502\n\u2502 identifier     \u2502 test--root--2023-10-13                                                                                                                \u2502\n\u2502 description    \u2502                                                                                                                                       \u2502\n\u2502 template_vars  \u2502 {'bar': 'BAR', 'baz': 'BAZ', 'foo': 'FOO', 'workdir': '/nfs'}                                                                         \u2502\n\u2502 template_files \u2502 [{'parent_id': 5, 'filename': 'dummy-script.py.j2', 'file_type': 'ENTRYPOINT', 'created_at': '2023-10-13T19:44:22.020542',            \u2502\n\u2502                \u2502 'updated_at': '2023-10-13T19:44:22.020556'}]                                                                                          \u2502\n\u2502 workflow_files \u2502 [{'parent_id': 5, 'filename': 'jobbergate.py', 'runtime_config': {'template_files': None, 'job_script_name': None,                    \u2502\n\u2502                \u2502 'default_template': 'dummy-script.py.j2', 'output_directory': '.', 'supporting_files': None, 'supporting_files_output_name': None},   \u2502\n\u2502                \u2502 'created_at': '2023-10-13T19:44:22.110739', 'updated_at': '2023-10-13T19:44:22.110750'}]                                              \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n

    Verify that the id, name, identifier, and timestamps match the Application that was created.

    Next, fetch the same application by identifier and verify that it is the same Application:

    jobbergate --full applications get-one --identifier=test--tbeck--2023-10-13\n
    "},{"location":"developer_guide/integration_testing/#updating-an-application","title":"Updating an Application","text":"

    Next, verify that you can update the application through the CLI.

    Run this command to verify that we can change the name:

    jobbergate applications update --id=1 --application-desc=\"Here is a test description\"\n

    Verify that you can see the new description in the application when you fetch it via the get-one subcommand. Also, check the output with the --full parameter to make sure that the updated_at field is different and later than the created_at field.

    "},{"location":"developer_guide/integration_testing/#rendering-a-job-script-from-an-application","title":"Rendering a Job Script from an Application","text":"

    Now that an application has been uploaded uploaded, use it to render a new Job Script.

    There are a few different options to test here to check for correct behavior:

    • Basic, interactive render
    • Render in \"fast mode\" with a --param-file
    • Render with additional SBATCH params
    "},{"location":"developer_guide/integration_testing/#basic-interactive-render","title":"Basic, interactive render","text":"

    First, render an Application to a Job Script by executing the interactive code that gathers the values for template variables from the user.

    To start the rendering process, execute:

    jobbergate job-scripts render --name=$NAME --application-id=1\n

    Verify that you are shown 3 prompts to supply values for the template variables. Fill these in with any values you like. Notice that the third question has a default response supplied already. Accept this value or replace it with your preferred value:

       [?] gimme the foo!: FOO\n   [?] gimme the bar!: BAR\n   [?] gimme the foo!: BAZ\n   [?] gimme the workdir!: /nfs\n

    When prompted if you would like to submit the job, decline with \"n\".

    After completing the questions, verify that the CLI reports that the new Job Script was created using the supplied values:

                     Created Job Script\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key            \u2503 Value                            \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 id             \u2502 1                                \u2502\n\u2502 application_id \u2502 1                                \u2502\n\u2502 name           \u2502 test--root--2023-10-13           \u2502\n\u2502 description    \u2502                                  \u2502\n\u2502 owner_email    \u2502 local-user@jobbergate.local-mail \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n

    Now we need to verify that the Job Script was rendered with the correct values. Run the following command:

    jobbergate job-scripts show-files --id=1\n

    The output should show the Job Script with the provided template variable values rendered as expected:

    \u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 dummy-script.py \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502                                                                                                                                                        \u2502\n\u2502   #!/bin/python3                                                                                                                                       \u2502\n\u2502                                                                                                                                                        \u2502\n\u2502   #SBATCH -J dummy_job                                                                                                                                 \u2502\n\u2502   #SBATCH -t 60                                                                                                                                        \u2502\n\u2502                                                                                                                                                        \u2502\n\u2502   print(\"Executing dummy job script\")                                                                                                                  \u2502\n\u2502   with open(\"/nfs/dummy-output.txt\", mode=\"w\") as dump_file:                                                                                           \u2502\n\u2502       print(\"I am a very, very dumb job script\", file=dump_file)                                                                                       \u2502\n\u2502       print(\"foo=FOO\", file=dump_file)                                                                                                                 \u2502\n\u2502       print(\"bar=BAR\", file=dump_file)                                                                                                                 \u2502\n\u2502       print(\"baz=BAZ\", file=dump_file)                                                                                                                 \u2502\n\u2502                                                                                                                                                        \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 This is the main job script file \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n
    "},{"location":"developer_guide/integration_testing/#render-in-fast-mode-with-a-param-file","title":"Render in \"fast mode\" with a --param-file","text":"

    Next, verify that a Job Script can be rendered while skipping the interactive question answering segment by pre-supplying the application with the values to use for rendering. Since only the third question has a default, supply at least the other two questions with a param using the --param-file parameter.

    First, create a file to hold the params (hit ctrl-d to finish and write the file):

    cat > params.json\n{\n  \"foo\": \"FOO\",\n  \"bar\": \"BAR\"\n}\n

    Now, render the Application using this file. Include the --no-submit flag because the Job Script shouldn't be submitted immediately. Verify only the rendering process for the new Job Script:

    jobbergate job-scripts render --name=$NAME --application-id=1 --fast --param-file=params.json --no-submit\n

    The output from the command will show you the default values that were used that you did not specify in the params.json file:

    Default values used\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key     \u2503 Value \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 baz     \u2502 zab   \u2502\n\u2502 workdir \u2502 /nfs  \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n\n\n                 Created Job Script\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key            \u2503 Value                            \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 id             \u2502 2                                \u2502\n\u2502 application_id \u2502 1                                \u2502\n\u2502 name           \u2502 test--root--2023-10-13           \u2502\n\u2502 description    \u2502                                  \u2502\n\u2502 owner_email    \u2502 local-user@jobbergate.local-mail \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n

    Now check the rendered file again using the show-files sub-command.

    "},{"location":"developer_guide/integration_testing/#render-with-additional-sbatch-params","title":"Render with additional SBATCH params","text":"

    Finally, test that additional SBATCH params can be inserted at render time. The code will insert these additional parameters into the rendered Job Script files.

    To supply extra SBATCH params, they are provided on the command line using the --sbatch-params option. Use this command to test it out:

    jobbergate job-scripts render --name=$NAME --application-id=1 --fast --param-file=params.json --no-submit --sbatch-params=\"--cluster=fake\" --sbatch-params=\"--partition=dummy\"\n

    The output should look like:

    Default values used\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key     \u2503 Value \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 baz     \u2502 zab   \u2502\n\u2502 workdir \u2502 /nfs  \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n\n\n                 Created Job Script\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key            \u2503 Value                            \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 id             \u2502 3                                \u2502\n\u2502 application_id \u2502 1                                \u2502\n\u2502 name           \u2502 test--root--2023-10-13           \u2502\n\u2502 description    \u2502                                  \u2502\n\u2502 owner_email    \u2502 local-user@jobbergate.local-mail \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n

    Now, review the rendered Job Script file using the --show-files command. You should see that the additional two SBATCH parameters are included:

    #SBATCH --cluster=fake                                                                                                                               \u2502\n#SBATCH --partition=dummy\n
    "},{"location":"developer_guide/integration_testing/#updating-a-job-script","title":"Updating a Job Script","text":"

    Next, verify that an existing Job Script can be updated.

    Run this command to verify that you can change the description:

    jobbergate job-scripts update --id=1 --description=\"Here is a test description\"\n

    Verify that you can see the new description in the Job Script when you fetch it via the get-one subcommand. Also, check the output with the --full parameter to make sure that the updated_at field is different and later than the created_at field.

    "},{"location":"developer_guide/integration_testing/#submitting-a-job-script","title":"Submitting a Job Script","text":"

    Next, test the process of submitting a Job Script to a slurm cluster for execution. Note that the docker-compose.yaml used for testing sets up a volume-mounted directory named /nfs. The /nfs directory in the container is mounted from the slurm-fake-nfs directory in the jobbergate-composed subproject. You can look in this directory after the job completes execution to check the results.

    You will need to verify that jobs are being submitted correctly vai the following steps:

    • Submit the job through the CLI
    • Verify that the agent submitted the job
    • Verify that the agent updates the Job Submission status
    • Verify the output from the job
    "},{"location":"developer_guide/integration_testing/#submit-the-job-through-the-cli","title":"Submit the job through the CLI","text":"

    Submit the Job Script using the CLI by running the following command:

    jobbergate job-submissions create --name=$NAME --job-script-id=1 --cluster-name=local-slurm --execution-directory=/nfs\n

    Verify that the output shows that the Job Submission has been created for the target Job Script

                      Created Job Submission\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Key                 \u2503 Value                            \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 id                  \u2502 1                                \u2502\n\u2502 job_script_id       \u2502 1                                \u2502\n\u2502 cluster_name        \u2502 local-slurm                      \u2502\n\u2502 slurm_job_id        \u2502 None                             \u2502\n\u2502 execution_directory \u2502 /nfs                             \u2502\n\u2502 name                \u2502 test--root--2023-10-13           \u2502\n\u2502 description         \u2502                                  \u2502\n\u2502 owner_email         \u2502 local-user@jobbergate.local-mail \u2502\n\u2502 status              \u2502 CREATED                          \u2502\n\u2502 report_message      \u2502 None                             \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n
    "},{"location":"developer_guide/integration_testing/#verify-that-the-agent-submitted-the-job","title":"Verify that the agent submitted the job","text":"

    To verify that the agent submitted the job correctly, review the log output from the agent.

    The agent performs the following process to complete a Job Submission with Slurm

    • Fetch a pending job from the API
    • Submit the job to slurm
    • Mark the job as submitted and add the slurm job id to it

    You can access the log data by running the following command in a terminal that has changed directory to the jobbergate-composed folder:

    docker-compose logs jobbergate-agent\n

    It may be useful to pipe the output to a text viewer like less.

    If the agent has successfully submitted the job, you should see some log lines that look like this (ellipses indicate omitted content):

    ...Retrieved 1 pending job submission...\n...Submitting pending job submission 1\n...\n...Submitting pending job submission 1 to slurm...\n...\n...Received slurm job id 1 for job submission 1\n...Marking job job_submission_id=22 as SUBMITTED (slurm_job_id=1)\n

    If you find those log lines, then the Agent has successfully submitted the job to slurm.

    "},{"location":"developer_guide/integration_testing/#verify-that-the-job-was-completed","title":"Verify that the job was completed","text":"

    To verify that the job completed successfully, review the log output from the agent.

    The agent performs the following process to complete Job Submissions

    • Fetch the submitted job from the API
    • Check the status of the job in slurm using the the slurm_job_id
    • If the job is completed, mark the Job Submission as COMPLETED

    You should look for log lines that look like this (ellipses indicate omitted content):

    ...Retrieved 1 active job submissions...\n...Fetching status of job_submission 1 from slurm\n...Fetching slurm job status for slurm job 1\n...\n...Status for slurm job 1 is job_id=1 job_state='COMPLETED'...\n...Updating job_submission with status=COMPLETED\n
    "},{"location":"developer_guide/integration_testing/#verify-the-output-from-the-job","title":"Verify the output from the job","text":"

    In the terminal where you are running Jobbergate CLI commands, you can check the /nfs directory to see the results. You should see three output files in the directory:

    • test--root--2023-10-13.out
    • test--root--2023-10-13.err
    • dummy-output.txt

    First, check the standard output from the script:

    cat /nfs/test--root--2023-10-13.out\n

    You should see a single line that says:

    Executing dummy job script\n

    The standard error from the script should be empty.

    The final file, dummy-output.txt, should contain the following content:

    I am a very, very dumb job script\nfoo=FOO\nbar=BAR\nbaz=BAZ\n
    "},{"location":"developer_guide/integration_testing/#conclusion","title":"Conclusion","text":"

    The process described in this document covers integration tests across the entire Jobbergate platform. These integration tests should be performed before new versions of Jobbergate are published and before Jobbergate is deployed to a new environment.

    "},{"location":"developer_guide/keycloak_setup/","title":"Setting up Keycloak for Jobbergate","text":"

    Jobbergate's security is provided by the Armasec package which should be compatible with any OIDC provider. However, the recommended provider is Keycloak.

    In this guide, we outline the steps to integrate an existing Keycloak instance (version 19.0.2 as used in this example) with Jobbergate to ensure a smooth user experience and enhanced security features.

    Although this tutorial focuses on integrating Keycloak with a locally deployed instance of Jobbergate, such as one housed in a Docker container via the jobbergate-composed sub-project, the procedures can be easily adapted to suit deployments on single-node Keycloak clusters or other complex configurations.

    "},{"location":"developer_guide/keycloak_setup/#create-a-new-realm","title":"Create a new Realm","text":"

    You have the option to utilize an existing realm for Jobbergate, but for a streamlined process, it's typically more advantageous to create a new realm specifically for your Jobbergate deployment.

    Once you're logged into the Keycloak interface, navigate and click the Add realm button, found beneath the Select realm dropdown menu on the left-hand side.

    For those using a local Jobbergate deployment, you should assign the Name as \"jobbergate-local\".

    You'll also need to specify a Frontend URL. Avoid using \"localhost\" because a valid domain is required for the redirection to function correctly. A suitable alternative is the .local special domain; this domain is ideal as it isn't subject to reservation on any DNS. For instance, your full Frontend URL would be http://keycloak.local:8080.

    The remaining realm settings can be left at their default configurations.

    "},{"location":"developer_guide/keycloak_setup/#setup-hostfile","title":"Setup Hostfile","text":"

    For the Keycloak admin UI to work correctly in a local deployment, it's essential to include the keycloak.local domain in your system\u2019s hostfile.

    For users on Linux or OSX, you can find this file at /etc/hosts. Windows users can locate it at c:\\windows\\system32\\drivers\\etc\\hosts.

    Editing this file requires administrative or sudo privileges.

    Upon accessing the file, append the following line and save your changes:

    127.0.0.1   keycloak.local\n

    This step ensures that the Frontend URL resolves correctly, facilitating seamless navigation and operation.

    "},{"location":"developer_guide/keycloak_setup/#create-a-new-client-for-the-cli","title":"Create a new Client for the CLI","text":"

    To facilitate login and JWT authentication for the Jobbergate CLI, it's essential to allocate a dedicated client.

    Begin by navigating to the Clients section, found on the left sidebar, and then proceed to click on the Create button located on the right.

    When adjusting the Client Protocol settings, select the openid-connect option. For the Client ID setting, which choosing an easy to identify name like \"jobbergate-cli\" is best even though this field can be any unique string.

    To ensure the CLI can utilize this client for login purposes, it's vital to activate the OAuth 2.0 Device Authorization Grant option.

    If you're working with a local deployment, simply input \"*\" in the Valid Redirect URIs section.

    "},{"location":"developer_guide/keycloak_setup/#add-roles","title":"Add Roles","text":"

    Next, we need to add the needed roles for Jobbergate endpoints. These represent the fine-grained permissions that are checked for each request to make sure that the user has permission to fulfill the request.

    Click the Roles tab at the top, and then click on Add Role on the right.

    Add the following roles:

    Name Description jobbergate:job-templates:edit Allow to view job templates jobbergate:job-templates:view Allow to view job templates jobbergate:job-scripts:edit Allow to edit job scripts jobbergate:job-scripts:view Allow to view job scripts jobbergate:job-submissions:edit Allow to edit job submissions jobbergate:job-submissions:view Allow to view job submissions"},{"location":"developer_guide/keycloak_setup/#add-mappers","title":"Add Mappers","text":"

    Jobbergate requires two claims that are not available by default. We will add them to the JWTs with Mappers.

    Click the Mappers tab at the top, and then click the Create button to add a new Mapper.

    "},{"location":"developer_guide/keycloak_setup/#audience","title":"Audience","text":"

    First, we need to add an \"audience\" mapper. Select \"audience\" for the Name field. Next, select \"Audience\" for the Mapper Type. The Included Custom Audience value may be whatever you like. The local deploy, by default, uses https://apis.omnivector.solutions. Make sure to enable the Add to ID token setting.

    "},{"location":"developer_guide/keycloak_setup/#permissions","title":"Permissions","text":"

    The Armasec package expects to find \"permissions\" in a claim at the root of the JWT payload. To facilitate this, we need to add a mapper that will copy the permissions to the correct place in the JWT. We will call the new mapper our \"permissions\" mapper.

    Enter \"Permissions\" under the Name field. Next, select \"User Client Role\" as the Mapper Type. Select \"jobbergatel-cli\" for the Client ID. The Token Claim Name must have the value \"permissions\". The Claim JSON Type field must be \"String\".

    "},{"location":"developer_guide/keycloak_setup/#create-a-new-client-for-the-agent","title":"Create a new Client for the Agent","text":"

    The Jobbergate Agent also requires its own client.

    Again, click the Clients section on the left navigation bar, and then click the Create button on the right.

    For the Client Protocol setting, choose the openid-connect protocol. The Client ID setting will be used to match jobs to the cluster they should be submitted to. So use the cluster name for this setting. For a local deployment, the Client ID should be \"local-slurm\".

    On the Settings tab, set Access Type to confidential and enter \"*\" for the Valid Redirect URIs. Scroll down and click on the Save button.

    "},{"location":"developer_guide/keycloak_setup/#add-roles_1","title":"Add Roles","text":"

    Click on the Roles tab, and click the Add Role button. Add all the following roles as above:

    Name Description jobbergate:job-templates:edit Allow to view Jobbergate applications jobbergate:job-templates:view Allow to view applications jobbergate:job-scripts:edit Allow to edit job scripts jobbergate:job-scripts:view Allow to view job scripts jobbergate:job-submissions:edit Allow to edit job submissions jobbergate:job-submissions:view Allow to view job submissions"},{"location":"developer_guide/keycloak_setup/#add-mappers_1","title":"Add Mappers","text":"

    Like the CLI client, the Agent's client also requires the \"Audience\" and \"Permissions\" mappers.

    Click the Mappers tab at the top, and then click the Create button to add a new Mapper.

    "},{"location":"developer_guide/keycloak_setup/#audience_1","title":"Audience","text":"

    First, we need to add an \"audience\" mapper. Select \"audience\" for the Name field. Next, select \"Audience\" for the Mapper Type. The Included Custom Audience value may be whatever you like. The local deploy, by default, uses \"https://apis.omnivector.solutions\". Make sure to enable the Add to ID token setting.

    "},{"location":"developer_guide/keycloak_setup/#permissions_1","title":"Permissions","text":"

    Next, add a \"permissions\" mapper. The Armasec package expects to find a \"permissions\" claims under a claim at the root of the JWT payload. Enter \"Permissions\" under the Name field. Next, select \"User Client Role\" as the Mapper Type. Select \"jobbergatel-cli\" for the Client ID. The Token Claim Name must have the value \"permissions\". The Claim JSON Type field must be \"String\".

    "},{"location":"developer_guide/keycloak_setup/#add-service-account-roles","title":"Add Service Account Roles","text":"

    To add the correct roles to the tokens issued for the Agent's client, we need to add some \"Service Account Roles\".

    Click the Service Account Roles tab. Then, from the Client Roles drop-down, select the local-slurm client. Select all of the Jobbergate roles created above and then click the Add selected button.

    "},{"location":"developer_guide/keycloak_setup/#create-users","title":"Create User(s)","text":"

    You will need to create some users that can use Jobbergate. These users will be able to sign-in through the Jobbergate CLI. Each user must have a unique email address. Other than that, no special settings are needed.

    To add a user, click Users on the left nav bar. Next, click the Add user button on the right.

    Use the following settings, and then click the Save button.

    | Username | local-user | | Email | local-user@jobbergate.local | | First Name | Local | | Last Name | User |

    After you have created the user, edit it by clicking on it in the list. You may need to click on the View all users button to see it.

    Click the Credentials tab at the top. Enter \"local\" for the Password and Password Confirmation field. Turn the Temporary setting to OFF, and click Reset Password. Click the Set password verification button.

    Next, click the Role Mappings tab at the top. Select the jobbergate-local entry in the Client Roles drop-down. Select all of the roles for jobbergate added above and click Add selected to add them to the user.

    "},{"location":"developer_guide/keycloak_setup/#conclusion","title":"Conclusion","text":"

    Your Keycloak instance is now prepared for use by Jobbergate! For additional information on configuring Keycloak and Armasec, consult documentation at:

    • https://www.keycloak.org/documentation
    • https://omnivector-solutions.github.io/armasec/
    "},{"location":"developer_guide/qa_tools/","title":"Quality Assurance Tools","text":"

    Jobbergate utilizes quality control tools across its three primary components (API, CLI, and Agent). The tools are invoked in the same way in each of the sub-projects, and may be invoked en masse from the root Jobbergate directory.

    "},{"location":"developer_guide/qa_tools/#running-unit-tests","title":"Running Unit Tests","text":"

    The main sub-projects each make use of pytest to apply unit testing. The unit tests for each are contained in a subdirectory named tests/.

    To invoke all of the unit tests for a sub-project, simply issue the following command:

    make test\n

    Once you enter the command above, the unit tests suite will start running. For the API, this process takes a few minutes. The others only take a few seconds. The status of the tests will be logged to the console as well as a coverage report for the unit tests:

    ================================================================== test session starts ===================================================================\nplatform linux -- Python 3.8.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0\nUsing --random-order-bucket=module\nUsing --random-order-seed=650699\n\nrootdir: /home/dusktreader/git-repos/omnivector/jobbergate/jobbergate-api, configfile: pyproject.toml, testpaths: jobbergate_api/tests\nplugins: asyncio-0.12.0, random-order-1.0.4, respx-0.17.1, env-0.6.2, armasec-0.11.0, freezegun-0.4.2, cov-2.12.1, anyio-3.5.0\ncollecting ... 2022-09-07 16:31:37.548 | INFO     | jobbergate_api.main:<module>:39 - Skipping Sentry\ncollected 158 items\n\ntests/apps/job_scripts/test_routers.py ........................                                                                     [ 15%]\ntests/apps/applications/test_schemas.py ....                                                                                        [ 17%]\ntests/test_file_validation.py ...........                                                                                           [ 24%]\ntests/test_email_notification.py .......                                                                                            [ 29%]\ntests/apps/applications/test_application_files.py .........                                                                         [ 34%]\ntests/apps/job_submissions/test_routers.py .................................                                                        [ 55%]\ntests/apps/job_scripts/test_job_script_files.py .........                                                                           [ 61%]\ntests/apps/test_main.py .                                                                                                           [ 62%]\ntests/test_meta_mapper.py ...                                                                                                       [ 63%]\ntests/test_s3_manager.py ...                                                                                                        [ 65%]\ntests/test_config.py ................                                                                                               [ 75%]\ntests/test_pagination.py ........                                                                                                   [ 81%]\ntests/test_storage.py ..                                                                                                            [ 82%]\ntests/test_security.py ...                                                                                                          [ 84%]\ntests/apps/applications/test_routers.py .........................                                                                   [100%]\n\n==================================================================== warnings summary ====================================================================\ntests/conftest.py:53\n  /home/dusktreader/git-repos/omnivector/jobbergate/jobbergate-api/tests/conftest.py:53: PytestUnknownMarkWarning: Unknown pytest.mark.enforce_empty_database - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/mark.html\n    @pytest.mark.enforce_empty_database()\n\ntests/apps/job_scripts/test_routers.py: 37 warnings\ntests/apps/job_submissions/test_routers.py: 40 warnings\ntests/test_pagination.py: 10 warnings\ntests/apps/applications/test_routers.py: 42 warnings\n  /home/dusktreader/.cache/pypoetry/virtualenvs/jobbergate-api-zc2JKxO9-py3.8/lib/python3.8/site-packages/databases/backends/postgres.py:114: DeprecationWarning: The `Row.keys()` method is deprecated to mimic SQLAlchemy behaviour, use `Row._mapping.keys()` instead.\n    warnings.warn(\n\n-- Docs: https://docs.pytest.org/en/stable/warnings.html\n\n---------- coverage: platform linux, python 3.8.12-final-0 -----------\nName                                                               Stmts   Miss  Cover   Missing\n------------------------------------------------------------------------------------------------\njobbergate_api/__init__.py                                             0      0   100%\njobbergate_api/apps/__init__.py                                        0      0   100%\njobbergate_api/apps/applications/__init__.py                           0      0   100%\njobbergate_api/apps/applications/application_files.py                 77      0   100%\njobbergate_api/apps/applications/models.py                             7      0   100%\njobbergate_api/apps/applications/routers.py                          136     14    90%   64-66, 129-131, 136-137, 210-212, 331-332, 341\njobbergate_api/apps/applications/schemas.py                           66      0   100%\njobbergate_api/apps/job_scripts/__init__.py                            0      0   100%\njobbergate_api/apps/job_scripts/job_script_files.py                  121      4    97%   67, 80-81, 270\njobbergate_api/apps/job_scripts/models.py                              7      0   100%\njobbergate_api/apps/job_scripts/routers.py                           132     11    92%   98-100, 108-109, 235-237, 267-269, 301\njobbergate_api/apps/job_scripts/schemas.py                            38      0   100%\njobbergate_api/apps/job_submissions/__init__.py                        0      0   100%\njobbergate_api/apps/job_submissions/constants.py                      11      0   100%\njobbergate_api/apps/job_submissions/models.py                          8      0   100%\njobbergate_api/apps/job_submissions/routers.py                       186     12    94%   101-103, 260-262, 382, 395-400, 406, 449\njobbergate_api/apps/job_submissions/schemas.py                        51      0   100%\njobbergate_api/apps/permissions.py                                     8      0   100%\njobbergate_api/config.py                                              58      1    98%   102\njobbergate_api/email_notification.py                                  28      0   100%\njobbergate_api/file_validation.py                                    102      6    94%   36-56, 111, 175\njobbergate_api/main.py                                                47      4    91%   31-37, 94\njobbergate_api/meta_mapper.py                                         24      1    96%   104\njobbergate_api/metadata.py                                             2      0   100%\njobbergate_api/pagination.py                                          31      0   100%\njobbergate_api/s3_manager.py                                          14      0   100%\njobbergate_api/security.py                                            22      0   100%\njobbergate_api/storage.py                                             52      1    98%   128\ntests/__init__.py                                       0      0   100%\ntests/apps/__init__.py                                  0      0   100%\ntests/apps/applications/__init__.py                     0      0   100%\ntests/apps/applications/test_application_files.py     104      0   100%\ntests/apps/applications/test_routers.py               368      0   100%\ntests/apps/applications/test_schemas.py                14      0   100%\ntests/apps/conftest.py                                 41      0   100%\ntests/apps/job_scripts/__init__.py                      0      0   100%\ntests/apps/job_scripts/conftest.py                     10      2    80%   32, 49\ntests/apps/job_scripts/test_job_script_files.py       102      0   100%\ntests/apps/job_scripts/test_routers.py                373      3    99%   48-64, 72\ntests/apps/job_submissions/__init__.py                  0      0   100%\ntests/apps/job_submissions/test_routers.py            483      0   100%\ntests/apps/test_main.py                                 7      0   100%\ntests/conftest.py                                     114      1    99%   127\ntests/test_config.py                                   33      0   100%\ntests/test_email_notification.py                       44      0   100%\ntests/test_file_validation.py                          17      0   100%\ntests/test_meta_mapper.py                              27      0   100%\ntests/test_pagination.py                               55      0   100%\ntests/test_s3_manager.py                               17      0   100%\ntests/test_security.py                                 39      0   100%\ntests/test_storage.py                                   7      0   100%\n------------------------------------------------------------------------------------------------\nTOTAL                                                               3083     60    98%\n\nRequired test coverage of 95.0% reached. Total coverage: 98.05%\n=========================================================== 158 passed, 130 warnings in 52.46s ===========================================================\n

    Note

    The API unit tests require that a test database is already running. You can start one by using the dev-tools provided in the API sub-project.

    "},{"location":"developer_guide/qa_tools/#running-linters","title":"Running Linters","text":"

    The main sub-projects each use a group of linting tools to make sure that the code follows code quality standards. These linters will report any lines or segments of the code that do not meet the project's standards.

    To invoke all of the linters for a sub-project, issue the following command:

    make lint\n

    If any issues are reported, fix the reported error and try running it again. The linters will only succeed if all of the issues are fixed.

    "},{"location":"developer_guide/qa_tools/#running-formatters","title":"Running Formatters","text":"

    For most of the linting issues, the code can be auto-corrected using the configured code formatters.

    Currently, the sub-projects use the following formatters:

    • black
    • isort

    To apply the formatters, use this command:

    make format\n

    The formatters will report any files that were changed in their reports.

    "},{"location":"developer_guide/qa_tools/#running-static-code-checkers","title":"Running Static Code Checkers","text":"

    The Jobbergate sub-projects include type-hints that must be checked using the mypy static code checker. It may invoked using make:

    make mypy\n

    If any issues are located, they will be reported. Each type issue must be fixed before the static type checker passes.

    "},{"location":"developer_guide/qa_tools/#running-all-quality-checks","title":"Running All Quality Checks","text":"

    Finally, all of the quality checks can be run using this command:

    make qa\n
    "},{"location":"developer_guide/template_workflows/","title":"Job Script Template Workflow Files","text":"

    The main workflow file is a python script that is used within an interactive framework that gathers the values for template variables that will be needed when Job Scripts are rendered from Applications.

    Throughout the documentation, this file is referred to as the \"Workflow Source File.\"

    The entire purpose of the Workflow Source File is to construct a workflow of questions organized in a series of that can be changed dynamically according to the answers provided by the user.

    "},{"location":"developer_guide/template_workflows/#the-jobbergateapplication-class","title":"The JobbergateApplication class","text":"

    Each Workflow Source File script must define exactly one class named JobbergateApplication.

    This class should be a regular python class that inherits from the JobbergateApplicationBase. This base class is imported from the application_base module.

    The JobbergateApplication implementation may be a simple or complex as needed by the user. However, it must define a mainflow() method which is the first of the workflow methods that the Application processes.

    "},{"location":"developer_guide/template_workflows/#the-workflow-methods","title":"The workflow methods","text":"

    The mainflow() method is essentially the entry point for the Workflow Source File. It must return a list of questions that should be asked to the user in order. These questions will be used to gather the template variable values.

    The mainflow() method must take a dictionary named data as a keyword argument. This kwarg should default to None, and it should be set to an empty dict if the default is not overridden.

    Each workflow can also specify the net workflow method to call after its questions have been asked and answered. In this way, the workflows can be organized in a dynamic series where the path is dictated by the user responses.

    The workflow methods specify the next flow in the sequence by setting an item keyed by \"nextworkflow\" in the data dictionary. The value of this item is the name of the next workflow method to call.

    Each workflow method can examine the results from previous workflows by referencing the data dict. All of the key/value pairs in the dictionary (besides \"nextworkflow\") represent answers to previous questions.

    "},{"location":"developer_guide/template_workflows/#the-questions","title":"The Questions","text":"

    The Workflow Source File is built around a question asking framework that defines different sorts of questions that can be asked of the user.

    The question types are defined by classes that derive from a base QuestionBase class. The question types include:

    • Text: gather a simple string response from the user
    • Integer: gather a simple int response from the user
    • List: prompt the user to select one item from a list
    • Directory: prompt the user for a directory path
    • File: prompt the user for a file path
    • Checkbox: prompt the user to select as many items from a list as they want
    • Confirm: prompt the user to offer a boolean response
    • BooleanList: prompt a series of boolean responses
    • Const: set the variable to the default value without even asking the user

    Note

    The BooleanList question has some very complex logic. The source code should be examined to understand what this does in detail.

    All of the implementation of the question classes (including the base class) can be found in the questions module of the Jobbergate source code.

    "},{"location":"developer_guide/template_workflows/#other-class-attributes","title":"Other class attributes","text":"

    Each Workflow Source File also has access to some attributes set up by the JobbergateApplicationBase.

    The jobbergate_config attribute will contain any of the properties that are set in the jobbergate_config section of the Application Config (jobbergate.yaml). These values can include anything set up by the user at application creation time.

    The application_config attribute contains all of the properties that are set in the application_config section of the Application config (jobbergate.yaml). This section may be empty. If it is, the application_config attribute will be an empty dictionary. This dictionary should only be populated by the template variables that the Workflow Source File seeks to collect from the user. The values for each item are the default values for that template variable.

    "},{"location":"elements/apps/","title":"Jobbergate Apps","text":"

    Jobbergate consists of three interconnected Python applications that operate harmoniously. These applications enable the creation and dispatch of Job Scripts to a Slurm cluster, eliminating the need for the Jobbergate user to engage directly with Slurm \u2013 a process that might be challenging or unfeasible.

    While the primary interface for user interaction with Jobbergate is the CLI, both the API and Core package can be employed to develop automation and craft tools leveraging Jobbergate's capabilities.

    The three apps in Jobbergate are:

    • Jobbergate API
    • Jobbergate CLI
    • Jobbergate Agent

    And the SDK that provides python integration is:

    • Jobbergate Core
    "},{"location":"elements/apps/agent/","title":"Jobbergate Agent Overview","text":"

    The Jobbergate Agent is a daemon application that is designed to be integrated into the slurm cluster.

    It predominantly fulfills two key roles:

    • Submitting newly created Job Submissions to the Slurm cluster
    • Monitoring and updating the status of Job Submissions as they undergo processing
    "},{"location":"elements/apps/agent/#submitting-jobs","title":"Submitting Jobs","text":"

    The Jobbergate Agent constantly monitors the Job Submissions resource for entries marked with a CREATED status. These are Job Submissions that the API has instantiated but are yet to be dispatched to Slurm.

    When submitting a job to Slurm, the Jobbergate Agent pulls the Job Script itself plus any supporting files associated with it down to the cluster. Once all the files have been downloaded, the Job Script is submitted to Slurm via it's RESTful API. The Job Submission saves the identifier for the Slurm Job so that it can be associated with the Job Script that was submitted. The Job Submission also tracks all of the supporting files and submission parameters that were submitted along with the Job Script.

    Upon job submission to Slurm, the Jobbergate Agent retrieves not only the Job Script but also any related supporting files, downloading them to the cluster. After ensuring all files are downloaded, the Job Script is dispatched to Slurm through its RESTful API. The Job Submission retains the unique identifier for the Slurm Job, ensuring it's linked to the submitted Job Script. Additionally, the Job Submission logs all the supporting files and submission parameters that were provided in tandem with the Job Script at submission time.

    "},{"location":"elements/apps/agent/#updating-job-status","title":"Updating Job Status","text":"

    Once submitted, the Jobbergate Agent updates the status of the Job Submission to SUBMITTED. If there is an error during the submission process, the Agent sets the Job Submission status to REJECTED.

    Upon completion of the job by the Slurm cluster, the Agent updates the status either to COMPLETE if successful, or FAILED if the job could not complete. This signifies the conclusion of tasks related to that particular Job Submission.

    "},{"location":"elements/apps/agent/#usage","title":"Usage","text":"

    The Jobbergate Agent operates in the background; it's designed to be initiated and left uninterrupted.

    For insights into its ongoing operations, the Agent offers detailed logging which can be analyzed.

    "},{"location":"elements/apps/api/","title":"Jobbergate API Overview","text":"

    The Jobbergate API is a RESTful API that functions as the Jobbergate platform's backbone. It offers access to the platform's data for various components, including the Jobbergate CLI, agent, and any other interfaces requiring interaction with Jobbergate assets.

    The API's endpoints are secured via OpenID Connect, and they require a valid auth token that is created when a user logs into the system.

    "},{"location":"elements/apps/api/#usage","title":"Usage","text":"

    The Jobbergate API is a standard RESTful API. It can be accessed vi a command-line tool like Curl or API testing tool like Postman.

    "},{"location":"elements/apps/api/#getting-an-auth-token","title":"Getting an Auth Token","text":"

    To use the Jobbergate API, you need to obtain an access token first. This token both authenticates your requests and provides authorization according to your user's rights.

    The authentication for Jobbergate API is managed by an affiliated OIDC service. While it's possible to directly interface with this service from your application to get an authentication token, the simplest method is via the Jobbergate CLI. Refer to the \"Logging In\" segment on the CLI usage page.

    "},{"location":"elements/apps/api/#querying-the-api","title":"Querying the API","text":"

    Once you have an auth token, you can interact with any of the Jobbergate API endpoints. The complete set of endpoints, parameters, and constraints are available through swagger documentation under jobbergate/docs wherever the API is deployed.

    For all requests made to secured endpoints, you must include the auth token in the Authorization header of your requests with a \"Bearer\" prefix.

    "},{"location":"elements/apps/api/#query-examples","title":"Query Examples","text":"

    To demonstrate how to use the API, the following examples will show how to fetch a list of all available Job Scripts.

    For these examples:

    • The auth token will be \"XXXXXXXX\"
    • The Jobbergate API is deployed at \"http://jobbergate.local\"
    "},{"location":"elements/apps/api/#curl","title":"curl","text":"

    From a linux terminal, you can use the curl command to make a request to the API:

    curl --header \"Authorization: Bearer XXXXXXXX\"  http://jobbergate.local/jobbergate/job-scripts\n

    The output of the above command should look something like: 

    {\"results\":[{\"id\":1,\"created_at\":\"2022-09-09T21:34:16.889289\",\"updated_at\":\"2022-09-09T21:34:16.889289\",\"job_script_name\":\"test script\",\"job_script_description\":null,\"job_script_owner_email\":\"local-user@jobbergate.local\",\"application_id\":1}],\"pagination\":{\"total\":1,\"start\":null,\"limit\":null}}\n

    To see the result more clearly, you can use a tool like jq to format the JSON response:

    {\n  \"results\": [\n    {\n      \"id\": 1,\n      \"created_at\": \"2022-09-09T21:34:16.889289\",\n      \"updated_at\": \"2022-09-09T21:34:16.889289\",\n      \"job_script_name\": \"test script\",\n      \"job_script_description\": null,\n      \"job_script_owner_email\": \"local-user@jobbergate.local\",\n      \"application_id\": 1\n    }\n  ],\n  \"pagination\": {\n    \"total\": 1,\n    \"start\": null,\n    \"limit\": null\n  }\n}\n
    "},{"location":"elements/apps/api/#python-and-httpx","title":"Python and httpx","text":"

    In python, you can use the httpx package to send requests to and process responses from the API:

    import json\nimport httpx\n\n\ntoken = \"XXXXXXXX\"\n\nresp = httpx.get(\n    \"http://localhost:8000/jobbergate/job-scripts\",\n    headers=dict(Authorization=f\"Bearer {token}\"),\n)\n\nprint(json.dumps(resp.json(), indent=2))\n

    The script will print out results like this:

    {\n  \"results\": [\n    {\n      \"id\": 1,\n      \"created_at\": \"2022-09-09T21:34:16.889289\",\n      \"updated_at\": \"2022-09-09T21:34:16.889289\",\n      \"job_script_name\": \"foo\",\n      \"job_script_description\": null,\n      \"job_script_owner_email\": \"local-user@jobbergate.local\",\n      \"application_id\": 1\n    }\n  ],\n  \"pagination\": {\n    \"total\": 1,\n    \"start\": null,\n    \"limit\": null\n  }\n}\n
    "},{"location":"elements/apps/cli/","title":"Jobbergate CLI Overview","text":"

    The Jobbergate CLI offers an interactive gateway to the functionalities of the Jobbergate API's. Users can utilize the CLI to manage resources and execute various tasks.

    The CLI operates under two primary modes:

    • Resource Creation: The CLI introduces create subcommands for every resource, allowing users to establish new instances.
    • Resource Viewing: With list and get-one subcommands available for each resource, users can inspect different detail levels about the resource entities stored in the database.

    To ensure secure access, the Jobbergate CLI offers a sign-in mechanism to the Jobbergate API. Once authenticated, users may use all the resources in Jobbergate that their account has been granted access to.

    "},{"location":"elements/apps/cli/#discovering-command-details","title":"Discovering Command details","text":"

    You can start learning about the commands and usage of the Jobbergate CLI by starting with this command:

    $ jobbergate --help\nUsage: jobbergate [OPTIONS] COMMAND [ARGS]...\n\n  Welcome to the Jobbergate CLI!\n\n  More information can be shown for each command listed below by running it\n  with the --help option.\n\nOptions:\n  --verbose / --no-verbose        Enable verbose logging to the terminal\n                                  [default: no-verbose]\n  --full / --no-full              Print all fields from CRUD commands\n                                  [default: no-full]\n  --raw / --no-raw                Print output from CRUD commands as raw json\n                                  [default: no-raw]\n  --version / --no-version        Print the version of jobbergate-cli and exit\n                                  [default: no-version]\n  --install-completion [bash|zsh|fish|powershell|pwsh]\n                                  Install completion for the specified shell.\n  --show-completion [bash|zsh|fish|powershell|pwsh]\n                                  Show completion for the specified shell, to\n                                  copy it or customize the installation.\n  --help                          Show this message and exit.\n\nCommands:\n  applications     Commands to interact with applications\n  job-scripts      Commands to interact with job scripts\n  job-submissions  Commands to interact with job submissions\n  login            Log in to the jobbergate-cli by storing the supplied...\n  logout           Logs out of the jobbergate-cli.\n  show-token       Show the token for the logged in user.\n

    If you want to delve deeper and understand the usage of a specific subcommand, you can use the --help flag with that particular subcommand. For example, to better understand the usage of the job-scripts create subcommand, you would run:

    $ jobbergate job-scripts create --help\n
    "},{"location":"elements/apps/cli/#logging-in","title":"Logging In","text":"

    The first thing you need to do with the Jobbergate CLI is to log in:

    jobbergate login\n
    Upon executing the command, a message will appear like:

    \u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Waiting for login \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502                                                                                                                      \u2502\n\u2502   To complete login, please open the following link in a browser:                                                    \u2502\n\u2502                                                                                                                      \u2502\n\u2502     http://keycloak.local:8080/realms/jobbergate-local/device?user_code=BMVJ-NLZS                                    \u2502\n\u2502                                                                                                                      \u2502\n\u2502   Waiting up to 5.0 minutes for you to complete the process...                                                       \u2502\n\u2502                                                                                                                      \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n\nWaiting for web login... \u2501\u257a\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501   3% 0:04:50\n

    Next, you will need to:

    1. Open the provided link by either clicking on it (if your terminal supports it) or copy/paste it into a browser.
    2. Enter your login credentials
    3. Complete the sign in process
    4. Return to your terminal

    You should see a message like:

    \u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Logged in! \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502                                                                                                                      \u2502\n\u2502   User was logged in with email 'local-user@jobbergate.local'                                                        \u2502\n\u2502                                                                                                                      \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n
    "},{"location":"elements/apps/cli/#checking-the-auth-token","title":"Checking the Auth Token","text":"

    To get access to the auth token you acquired by logging in, run this command:

    jobbergate show-token --plain\n
    Executing this command will display the authentication token in a plain text format, without any additional characters or formatting. This makes it easier for you to manually select and copy the token, especially in environments where clipboard access might be restricted, such as when using docker-compose or an SSH connection.

    Once the token is displayed, you can copy the token to your clipboard to use with API requests.

    It's essential to treat this token with care, as it provides access to the Jobbergate system under your user account. Ensure you don't share it with unauthorized individuals and avoid unintentionally exposing it in logs or scripts.

    "},{"location":"elements/apps/cli/#resource-commands","title":"Resource Commands","text":"

    Now that you are logged in, you can interact with any of the three main Jobbergate resources. Most of the resources provide the following sub-commands:

    • create: Create a new instance of the resource
    • delete: Delete an instance of the resource
    • get-one: Fetch details about a single instance of the resource
    • list: Fetch a listing of all the resources limited by filters
    • update: Update an instance of the resource.

    Details for each subcommand can be viewed by passing the --help flag to any of them.

    Use the --help option to explore the CLI and disccover the usage and options for all the subcommands.

    "},{"location":"elements/apps/cli/#usability","title":"Usability","text":"

    When rendering a job-script from a template, the user will be asked a series of questions to fill in the template variables.

    The library used for the questionnaire has a limitation that messages can only be displayed in a single line. This means that some of the questions can be truncated and will not be fully visible if the message is too long.

    Note

    To ensure that you can see the full output of the CLI, we recommend that you use a terminal in a maximized window.

    "},{"location":"elements/apps/core/","title":"Jobbergate Core Overview","text":"

    Jobbergate-core is an SDK designed to offer seamless access to Jobbergate's features within any Python project. It is ideal for constructing automation atop the Jobbergate platform or tailoring workflows that hinge on Jobbergate components.

    Coming Soon: More in-depth documentation of Jobbergate Core

    "},{"location":"elements/resources/","title":"Jobbergate Resources","text":"

    Jobbergate utilizes three primary resources for the efficient management of job script creation, templating, and submission. These resources are maintained in distinct database tables and can be accessed via individual API endpoints or through specific subcommands in the CLI.

    The principal resources of Jobbergate include:

    • Job Scripts
    • Job Script Templates
    • Job Submissions
    "},{"location":"elements/resources/job_script_templates/","title":"Job Script Templates","text":"

    Job Script Templates serve as adaptable blueprints for Job Scripts, allowing for the dynamic replacement of crucial values upon rendering. The end result of this process is a Job Script primed for cluster submission.

    The specific values incorporated into the template to generate a Job Script are termed \"template variables.\" Users can define constrains and default settings for these values within the Job Script Template's workflow script.

    Additionally, Job Script Templates provide a framework that allows for the interactive collection of values from users via the Jobbergate CLI.

    "},{"location":"elements/resources/job_scripts/","title":"Job Scripts","text":"

    In Jobbergate, the primary resource is the Job Script. These scripts dictate the instructions for jobs intended to execute on the Slurm cluster. They can either be Python files or shell scripts. Jobbergate facilitates the generation, modification, and submission of these Job Scripts to the cluster.

    Job Scripts can either be uploaded directly from a user's workstation or be derived by rendering the Job Script Templates.

    Submission of Job Scripts to any affiliated Slurm cluster can be accomplished through the CLI, API, or Core integrations. After submission, the execution status of a Job Script can be monitored using the Job Submission resource.

    "},{"location":"elements/resources/job_submissions/","title":"Job Submissions","text":""},{"location":"elements/resources/job_submissions/#job-submissions","title":"Job Submissions","text":"

    Job Submissions primarily monitor the status and metadata of a Job Script dispatched by Jobbergate to a Slurm cluster. They possess identifying details linking them to the Job Script that was submitted and to the corresponding Job objects created by Slurm.

    "},{"location":"reference/agent/","title":"Jobbergate Agent Reference","text":""},{"location":"reference/agent/#jobbergate_agent","title":"jobbergate_agent","text":""},{"location":"reference/agent/#jobbergate_agent.clients","title":"clients","text":""},{"location":"reference/agent/#jobbergate_agent.clients.cluster_api","title":"cluster_api","text":"

    Core module for Jobbergate API clients management

    "},{"location":"reference/agent/#jobbergate_agent.clients.cluster_api.AsyncBackendClient","title":"AsyncBackendClient","text":"

    Bases: AsyncClient

    Extends the httpx.AsyncClient class with automatic token acquisition for requests. The token is acquired lazily on the first httpx request issued. This client should be used for most agent actions.

    "},{"location":"reference/agent/#jobbergate_agent.clients.cluster_api.acquire_token","title":"acquire_token","text":"
    acquire_token(token: Token) -> Token\n

    Retrieves a token from OIDC based on the app settings.

    "},{"location":"reference/agent/#jobbergate_agent.clients.slurmrestd","title":"slurmrestd","text":"

    Core module for Jobbergate API clients management

    "},{"location":"reference/agent/#jobbergate_agent.clients.slurmrestd.AsyncBackendClient","title":"AsyncBackendClient","text":"

    Bases: AsyncClient

    Extends the httpx.AsyncClient class with automatic token acquisition for requests. The token is acquired lazily on the first httpx request issued. This client should be used for most agent actions.

    "},{"location":"reference/agent/#jobbergate_agent.clients.slurmrestd.acquire_token","title":"acquire_token","text":"
    acquire_token(username: str) -> str\n

    Retrieves a token from Slurmrestd based on the app settings.

    "},{"location":"reference/agent/#jobbergate_agent.clients.slurmrestd.inject_token","title":"inject_token","text":"
    inject_token(\n    request: httpx.Request,\n    username: typing.Optional[str] = None,\n) -> httpx.Request\n

    Inject a token based on the provided username into the request.

    For requests that need to use something except the default username, this injector should be used at the request level (instead of at client initialization) like this:

    .. code-block:: python

    client.get(url, auth=lambda r: inject_token(r, username=username))

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate","title":"jobbergate","text":""},{"location":"reference/agent/#jobbergate_agent.jobbergate.api","title":"api","text":""},{"location":"reference/agent/#jobbergate_agent.jobbergate.api.SubmissionNotifier","title":"SubmissionNotifier dataclass","text":"

    Class used to update the status for a job submission when some error is detected.

    It is designed to work together with py-buzz, extracting the error message, logging it and sending it to Jobbergate API.

    report_error async 
    report_error(params: DoExceptParams) -> None\n

    Update the status for a job submission.

    :param DoExceptParams params: Dataclass for the do_except user supplied handling method.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.api.fetch_active_submissions","title":"fetch_active_submissions async","text":"
    fetch_active_submissions() -> list[ActiveJobSubmission]\n

    Retrieve a list of active job_submissions.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.api.fetch_pending_submissions","title":"fetch_pending_submissions async","text":"
    fetch_pending_submissions() -> list[PendingJobSubmission]\n

    Retrieve a list of pending job_submissions.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.api.mark_as_submitted","title":"mark_as_submitted async","text":"
    mark_as_submitted(\n    job_submission_id: int, slurm_job_id: int\n)\n

    Mark job_submission as submitted in the Jobbergate API.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.api.update_status","title":"update_status async","text":"
    update_status(\n    job_submission_id: int,\n    status: JobSubmissionStatus,\n    *,\n    report_message: str | None = None\n) -> None\n

    Update a job submission with a status

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.constants","title":"constants","text":""},{"location":"reference/agent/#jobbergate_agent.jobbergate.constants.FileType","title":"FileType","text":"

    Bases: str, Enum

    File type enum.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.constants.JobSubmissionStatus","title":"JobSubmissionStatus","text":"

    Bases: str, Enum

    Enumeration of possible job_submission statuses.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.finish","title":"finish","text":""},{"location":"reference/agent/#jobbergate_agent.jobbergate.finish.finish_active_jobs","title":"finish_active_jobs async","text":"
    finish_active_jobs()\n

    Mark all active jobs that have completed or failed as finished.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.schemas","title":"schemas","text":""},{"location":"reference/agent/#jobbergate_agent.jobbergate.schemas.ActiveJobSubmission","title":"ActiveJobSubmission","text":"

    Bases: BaseModel

    Specialized model for the cluster-agent to pull an active job_submission.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.schemas.JobScript","title":"JobScript","text":"

    Bases: BaseModel

    Model to match database for the JobScript resource.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.schemas.JobScriptFile","title":"JobScriptFile","text":"

    Bases: BaseModel

    Model for the job_script_files field of the JobScript resource.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.schemas.PendingJobSubmission","title":"PendingJobSubmission","text":"

    Bases: BaseModel

    Specialized model for the cluster-agent to pull a pending job_submission along with data from its job_script and application sources.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.schemas.SlurmJobParams","title":"SlurmJobParams","text":"

    Bases: BaseModel

    Specialized model for describing job submission parameters for Slurm REST API.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.schemas.SlurmJobSubmission","title":"SlurmJobSubmission","text":"

    Bases: BaseModel

    Specialized model for describing a request to submit a job to Slurm REST API.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.schemas.SlurmSubmitError","title":"SlurmSubmitError","text":"

    Bases: BaseModel

    Specialized model for error content in a SlurmSubmitResponse.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.schemas.SlurmSubmitResponse","title":"SlurmSubmitResponse","text":"

    Bases: BaseModel

    Specialized model for the cluster-agent to pull a pending job_submission along with data from its job_script and application sources.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.schemas.SlurmSubmittedJobStatus","title":"SlurmSubmittedJobStatus","text":"

    Bases: BaseModel

    Specialized model for the cluster-agent to pull a concluded job_submission.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.submit","title":"submit","text":""},{"location":"reference/agent/#jobbergate_agent.jobbergate.submit.get_job_parameters","title":"get_job_parameters","text":"
    get_job_parameters(\n    slurm_parameters: Dict[str, Any], **kwargs\n) -> SlurmJobParams\n

    Obtain the job parameters from the slurm_parameters dict and additional values.

    Extra keyword arguments can be used to supply default values for any parameter (like name or current_working_directory). Note they may be overwritten by values from slurm_parameters.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.submit.get_job_script_file","title":"get_job_script_file async","text":"
    get_job_script_file(\n    pending_job_submission: PendingJobSubmission,\n    submit_dir: Path,\n) -> str\n

    Get the job script file from the backend.

    Write the job script file to the submit_dir if WRITE_SUBMISSION_FILES is set to True.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.submit.process_supporting_files","title":"process_supporting_files async","text":"
    process_supporting_files(\n    pending_job_submission: PendingJobSubmission,\n    submit_dir: Path,\n)\n

    Process the submission support files.

    Write the support files to the submit_dir if WRITE_SUBMISSION_FILES is set to True. Reject the submission if there are support files with WRITE_SUBMISSION_FILES set to False.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.submit.retrieve_submission_file","title":"retrieve_submission_file async","text":"
    retrieve_submission_file(file: JobScriptFile) -> str\n

    Get a submission file from the backend and return the decoded file content.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.submit.submit_job_script","title":"submit_job_script async","text":"
    submit_job_script(\n    pending_job_submission: PendingJobSubmission,\n    user_mapper: SlurmUserMapper,\n) -> int\n

    Submit a Job Script to slurm via the Slurm REST API.

    :param: pending_job_submission: A job_submission with fields needed to submit. :returns: The slurm_job_id for the submitted job

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.submit.submit_pending_jobs","title":"submit_pending_jobs async","text":"
    submit_pending_jobs()\n

    Submit all pending jobs and update them with SUBMITTED status and slurm_job_id.

    :returns: The slurm_job_id for the submitted job

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.submit.unpack_error_from_slurm_response","title":"unpack_error_from_slurm_response","text":"
    unpack_error_from_slurm_response(\n    response: SlurmSubmitResponse,\n) -> str\n

    Unpack the error message from the response of a slurmrestd request.

    "},{"location":"reference/agent/#jobbergate_agent.jobbergate.submit.write_submission_file","title":"write_submission_file async","text":"
    write_submission_file(\n    file_content: str, filename: str, submit_dir: Path\n)\n

    Write a decoded file content to the submit_dir.

    "},{"location":"reference/agent/#jobbergate_agent.main","title":"main","text":""},{"location":"reference/agent/#jobbergate_agent.settings","title":"settings","text":""},{"location":"reference/agent/#jobbergate_agent.settings.Settings","title":"Settings","text":"

    Bases: BaseSettings

    "},{"location":"reference/agent/#jobbergate_agent.settings.Settings.Config","title":"Config","text":"

    Provide configuration for the project settings.

    Note that we disable use of dotenv if we are in test mode.

    "},{"location":"reference/agent/#jobbergate_agent.settings.Settings.compute_extra_settings","title":"compute_extra_settings","text":"
    compute_extra_settings(values)\n

    Compute settings values that are based on other settings values.

    "},{"location":"reference/agent/#jobbergate_agent.tasks","title":"tasks","text":"

    Task definitions for the Jobbergate Agent.

    "},{"location":"reference/agent/#jobbergate_agent.tasks.active_submissions_task","title":"active_submissions_task","text":"
    active_submissions_task(scheduler: BaseScheduler) -> Job\n

    Schedule a task to handle active jobs every TASK_JOBS_INTERVAL_SECONDS seconds.

    "},{"location":"reference/agent/#jobbergate_agent.tasks.garbage_collection_task","title":"garbage_collection_task","text":"
    garbage_collection_task(\n    scheduler: BaseScheduler,\n) -> Union[Job, None]\n

    Schedule a task to perform garbage collection every dat at.

    "},{"location":"reference/agent/#jobbergate_agent.tasks.pending_submissions_task","title":"pending_submissions_task","text":"
    pending_submissions_task(scheduler: BaseScheduler) -> Job\n

    Schedule a task to submit pending jobs every TASK_JOBS_INTERVAL_SECONDS seconds.

    "},{"location":"reference/agent/#jobbergate_agent.tasks.trigger_garbage_collections","title":"trigger_garbage_collections async","text":"
    trigger_garbage_collections(\n    interval_between_calls: int = 60,\n) -> None\n

    Trigger maintenance tasks on the Jobbergate API.

    "},{"location":"reference/agent/#jobbergate_agent.utils","title":"utils","text":""},{"location":"reference/agent/#jobbergate_agent.utils.exception","title":"exception","text":"

    Core module for exception related operations

    "},{"location":"reference/agent/#jobbergate_agent.utils.exception.AuthTokenError","title":"AuthTokenError","text":"

    Bases: ClusterAgentError

    Raise exception when there are connection issues with the backend

    "},{"location":"reference/agent/#jobbergate_agent.utils.exception.ClusterAgentError","title":"ClusterAgentError","text":"

    Bases: Buzz

    Raise exception when execution command returns an error

    "},{"location":"reference/agent/#jobbergate_agent.utils.exception.JobSubmissionError","title":"JobSubmissionError","text":"

    Bases: ClusterAgentError

    Raise exception when a job cannot be submitted raises any error

    "},{"location":"reference/agent/#jobbergate_agent.utils.exception.JobbergateApiError","title":"JobbergateApiError","text":"

    Bases: ClusterAgentError

    Raise exception when communication with Jobbergate API fails

    "},{"location":"reference/agent/#jobbergate_agent.utils.exception.ProcessExecutionError","title":"ProcessExecutionError","text":"

    Bases: ClusterAgentError

    Raise exception when execution command returns an error

    "},{"location":"reference/agent/#jobbergate_agent.utils.exception.SlurmParameterParserError","title":"SlurmParameterParserError","text":"

    Bases: ClusterAgentError

    Raise exception when Slurm mapper or SBATCH parser face any error

    "},{"location":"reference/agent/#jobbergate_agent.utils.exception.SlurmrestdError","title":"SlurmrestdError","text":"

    Bases: ClusterAgentError

    Raise exception when slurmrestd raises any error

    "},{"location":"reference/agent/#jobbergate_agent.utils.exception.handle_errors_async","title":"handle_errors_async async","text":"
    handle_errors_async(\n    message: str,\n    raise_exc_class: Union[\n        Type[Exception], None\n    ] = Exception,\n    raise_args: Optional[Iterable[Any]] = None,\n    raise_kwargs: Optional[Mapping[str, Any]] = None,\n    handle_exc_class: Union[\n        Type[Exception], Tuple[Type[Exception], ...]\n    ] = Exception,\n    do_finally: Callable[[], None] = noop,\n    do_except: Callable[[DoExceptParams], None] = noop,\n    do_else: Callable[[], None] = noop,\n) -> Iterator[None]\n

    Async context manager that will intercept exceptions and repackage them with a message attached.

    Example:

    .. code-block:: python

    with handle_errors(\"It didn't work\"): some_code_that_might_raise_an_exception()

    :param: message: The message to attach to the raised exception. :param: raise_exc_class: The exception type to raise with the constructed message if an exception is caught in the managed context.

                           Defaults to Exception.\n\n                       If ``None`` is passed, no new exception will be raised and only the\n                       ``do_except``, ``do_else``, and ``do_finally``\n                       functions will be called.\n

    :param: raise_args: Additional positional args (after the constructed message) that will passed when raising an instance of the raise_exc_class. :param: raise_kwargs: Keyword args that will be passed when raising an instance of the raise_exc_class. :param: handle_exc_class: Limits the class of exceptions that will be intercepted Any other exception types will not be caught and re-packaged. Defaults to Exception (will handle all exceptions). May also be provided as a tuple of multiple exception types to handle. :param: do_finally: A function that should always be called at the end of the block. Should take no parameters. :param: do_except: A function that should be called only if there was an exception. Must accept one parameter that is an instance of the DoExceptParams dataclass. Note that the do_except method is passed the original exception. :param: do_else: A function that should be called only if there were no exceptions encountered.

    "},{"location":"reference/agent/#jobbergate_agent.utils.logging","title":"logging","text":"

    Core module for logging operations

    "},{"location":"reference/agent/#jobbergate_agent.utils.logging.log_error","title":"log_error","text":"
    log_error(params: DoExceptParams)\n

    Provide a utility function to log a Buzz-based exception and the stack-trace of the error's context.

    :param: params: A DoExceptParams instance containing the original exception, a message describing it, and the stack trace of the error.

    "},{"location":"reference/agent/#jobbergate_agent.utils.logging.logger_wraps","title":"logger_wraps","text":"
    logger_wraps(\n    *,\n    entry: bool = True,\n    exit: bool = True,\n    level: str = \"DEBUG\"\n)\n

    Decorator to wrap a function with logging statements.

    Reference

    https://loguru.readthedocs.io/en/stable/resources/recipes.html

    "},{"location":"reference/agent/#jobbergate_agent.utils.plugin","title":"plugin","text":"

    Provide to the agent the ability to load custom plugins that are installed on the same environment.

    "},{"location":"reference/agent/#jobbergate_agent.utils.plugin.load_plugins","title":"load_plugins","text":"
    load_plugins(plugin_name: str) -> Dict[str, Any]\n

    Discover and load plugins available to the agent, allowing for third party ones to be included.

    Notice the ones shipped with the agent are also declared on the pyproject.toml file as plugins, even though they could be easily loaded directly from source. This aims to support tests and to demonstrate how to use the plugin system.

    Reference

    https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins/

    "},{"location":"reference/agent/#jobbergate_agent.utils.scheduler","title":"scheduler","text":"

    Provide the task scheduler for the agent and the main loop to run it.

    Custom tasks can be added to the agent as installable plugins, which are discovered at runtime.

    References

    https://github.com/agronholm/apscheduler https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins

    "},{"location":"reference/agent/#jobbergate_agent.utils.scheduler.JobbergateTask","title":"JobbergateTask","text":"

    Bases: Protocol

    Protocol to be implemented by any task that is expected to run on the scheduler.

    __call__ 
    __call__(scheduler: BaseScheduler) -> Union[Job, None]\n

    Specify a callable used to schedule a task and return the resulting job.

    This is handled to client code to give them the opportunity to handle their own configuration and to access the rich flexibility of the scheduler API.

    None can also be returned if no task is going to be scheduled due to internal business logic.

    "},{"location":"reference/agent/#jobbergate_agent.utils.scheduler.init_scheduler","title":"init_scheduler","text":"
    init_scheduler() -> BaseScheduler\n

    Initialize the scheduler and schedule all tasks.

    "},{"location":"reference/agent/#jobbergate_agent.utils.scheduler.schedule_tasks","title":"schedule_tasks","text":"
    schedule_tasks(scheduler: BaseScheduler) -> None\n

    Discovery and schedule all tasks to be run by the agent.

    "},{"location":"reference/agent/#jobbergate_agent.utils.scheduler.shut_down_scheduler","title":"shut_down_scheduler","text":"
    shut_down_scheduler(\n    scheduler: BaseScheduler, wait: bool = True\n) -> None\n

    Shutdown the scheduler.

    "},{"location":"reference/agent/#jobbergate_agent.utils.user_mapper","title":"user_mapper","text":"

    Provide to the agent a way to map email addresses from Jobbergate local Slurm users.

    Custom mappers can be added to the agent as installable plugins, which are discovered at runtime.

    "},{"location":"reference/agent/#jobbergate_agent.utils.user_mapper.SlurmUserMapper","title":"SlurmUserMapper module-attribute","text":"
    SlurmUserMapper = Mapping[str, str]\n

    Slurm user mappers are mappings from email addresses to local Slurm users.

    "},{"location":"reference/agent/#jobbergate_agent.utils.user_mapper.SingleUserMapper","title":"SingleUserMapper dataclass","text":"

    Bases: Mapping

    A user mapper that always returns the same user.

    __post_init__ 
    __post_init__()\n

    Validate the user mapper by asserting it is not an empty string.

    "},{"location":"reference/agent/#jobbergate_agent.utils.user_mapper.SlurmUserMapperFactory","title":"SlurmUserMapperFactory","text":"

    Bases: Protocol

    Protocol to be implemented by plugins on client code.

    A callable with no arguments is expected in order to handle to client code the configuration and initialization of any custom user mapper. Any object that implements the Mapping protocol can be returned.

    __call__ 
    __call__() -> SlurmUserMapper\n

    Specify the signature to build a user mapper.

    "},{"location":"reference/agent/#jobbergate_agent.utils.user_mapper.manufacture","title":"manufacture","text":"
    manufacture() -> SlurmUserMapper\n

    Create an instance of a Slurm user mapper given the app configuration.

    "},{"location":"reference/api/","title":"Jobbergate API Reference","text":""},{"location":"reference/api/#jobbergate_api","title":"jobbergate_api","text":"

    Main components of the application: routers, config, main, pagination and create super user script.

    "},{"location":"reference/api/#jobbergate_api.apps","title":"apps","text":"

    Resources of the API.

    "},{"location":"reference/api/#jobbergate_api.apps.constants","title":"constants","text":"

    Constants to be shared by all models.

    "},{"location":"reference/api/#jobbergate_api.apps.constants.FileType","title":"FileType","text":"

    Bases: str, Enum

    File type enum.

    "},{"location":"reference/api/#jobbergate_api.apps.dependencies","title":"dependencies","text":"

    Router dependencies shared for multiple resources.

    Note

    The dependencies can be reused multiple times, since FastAPI caches the results.

    "},{"location":"reference/api/#jobbergate_api.apps.dependencies.CrudServices","title":"CrudServices","text":"

    Bases: NamedTuple

    Provide a container class for the CRUD services.

    "},{"location":"reference/api/#jobbergate_api.apps.dependencies.FileServices","title":"FileServices","text":"

    Bases: NamedTuple

    Provide a container class for the file services.

    "},{"location":"reference/api/#jobbergate_api.apps.dependencies.SecureService","title":"SecureService dataclass","text":"

    Bases: SecureSession

    Dataclass to hold the secure session and the bucket.

    "},{"location":"reference/api/#jobbergate_api.apps.dependencies.Services","title":"Services","text":"

    Bases: NamedTuple

    Provide a container class for the services.

    "},{"location":"reference/api/#jobbergate_api.apps.dependencies.get_bucket_name","title":"get_bucket_name","text":"
    get_bucket_name(\n    override_bucket_name: str | None = None,\n) -> str\n

    Get the bucket name based on the environment.

    The name can be overridden when multi tenancy is enabled by passing a bucket name.

    "},{"location":"reference/api/#jobbergate_api.apps.dependencies.get_bucket_url","title":"get_bucket_url","text":"
    get_bucket_url() -> str | None\n

    Get the bucket url based on the environment.

    "},{"location":"reference/api/#jobbergate_api.apps.dependencies.s3_bucket","title":"s3_bucket async","text":"
    s3_bucket(\n    bucket_name: str, s3_url: str | None\n) -> AsyncIterator[Bucket]\n

    Create a bucket using a context manager.

    "},{"location":"reference/api/#jobbergate_api.apps.dependencies.secure_services","title":"secure_services","text":"
    secure_services(\n    *scopes: str,\n    permission_mode: PermissionMode = PermissionMode.ALL,\n    commit: bool = True,\n    ensure_email: bool = False,\n    ensure_client_id: bool = False\n)\n

    Dependency to bind database services to a secure session.

    "},{"location":"reference/api/#jobbergate_api.apps.dependencies.service_factory","title":"service_factory","text":"
    service_factory(\n    session: AsyncSession, bucket: Bucket\n) -> Iterator[Services]\n

    Create the services and bind them to a db section and s3 bucket.

    "},{"location":"reference/api/#jobbergate_api.apps.garbage_collector","title":"garbage_collector","text":"

    Delete unused files from jobbergate's file storage.

    "},{"location":"reference/api/#jobbergate_api.apps.garbage_collector.delete_files_from_bucket","title":"delete_files_from_bucket async","text":"
    delete_files_from_bucket(\n    bucket, files_to_delete: set[str]\n) -> None\n

    Delete files from the bucket.

    "},{"location":"reference/api/#jobbergate_api.apps.garbage_collector.garbage_collect","title":"garbage_collect async","text":"
    garbage_collect(\n    session,\n    bucket,\n    list_of_tables,\n    background_tasks: BackgroundTasks,\n) -> None\n

    Delete unused files from jobbergate's file storage.

    "},{"location":"reference/api/#jobbergate_api.apps.garbage_collector.get_files_to_delete","title":"get_files_to_delete async","text":"
    get_files_to_delete(session, table, bucket) -> set[str]\n

    Get a set of files to delete.

    "},{"location":"reference/api/#jobbergate_api.apps.garbage_collector.get_set_of_files_from_bucket","title":"get_set_of_files_from_bucket async","text":"
    get_set_of_files_from_bucket(bucket, table) -> set[str]\n

    Get a set of files from the bucket.

    "},{"location":"reference/api/#jobbergate_api.apps.garbage_collector.get_set_of_files_from_database","title":"get_set_of_files_from_database async","text":"
    get_set_of_files_from_database(session, table) -> set[str]\n

    Get a set of files from the database.

    "},{"location":"reference/api/#jobbergate_api.apps.job_script_templates","title":"job_script_templates","text":"

    Module for the job script templates.

    "},{"location":"reference/api/#jobbergate_api.apps.job_script_templates.constants","title":"constants","text":"

    Describe constants for the job script templates module.

    "},{"location":"reference/api/#jobbergate_api.apps.job_script_templates.models","title":"models","text":"

    Database models for the job_script_templates resource.

    JobScriptTemplate

    Bases: CrudMixin, Base

    Job script template table definition.

    Notice all relationships are lazy=\"raise\" to prevent n+1 implicit queries. This means that the relationships must be explicitly eager loaded using helper functions in the class.

    Attributes:

    Name Type Description identifier Mapped[Optional[str]]

    The identifier of the job script template.

    template_vars Mapped[dict[str, Any]]

    The template variables of the job script template.

    See Mixin class definitions for other columns.

    include_files classmethod 
    include_files(query: Select) -> Select\n

    Include custom options on a query to eager load files.

    searchable_fields classmethod 
    searchable_fields()\n

    Add identifier as a searchable field.

    sortable_fields classmethod 
    sortable_fields()\n

    Add identifier as a sortable field.

    JobScriptTemplateFile

    Bases: FileMixin, Base

    Job script template files table definition.

    Attributes:

    Name Type Description parent_id Mapped[int]

    A foreign key to the parent job script template row.

    file_type Mapped[FileType]

    The type of the file.

    See Mixin class definitions for other columns

    WorkflowFile

    Bases: FileMixin, Base

    Workflow file table definition.

    Attributes:

    Name Type Description parent_id Mapped[int]

    A foreign key to the parent job script template row.

    runtime_config Mapped[dict[str, Any]]

    The runtime configuration of the workflow.

    See Mixin class definitions for other columns

    "},{"location":"reference/api/#jobbergate_api.apps.job_script_templates.routers","title":"routers","text":"

    Router for the Job Script Template resource.

    job_script_template_create async 
    job_script_template_create(\n    create_request: JobTemplateCreateRequest,\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_EDIT,\n            ensure_email=True,\n        )\n    ),\n)\n

    Create a new job script template.

    job_script_template_delete async 
    job_script_template_delete(\n    id_or_identifier: int | str = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_EDIT,\n            ensure_email=True,\n        )\n    ),\n)\n

    Delete a job script template by id or identifier.

    job_script_template_delete_file async 
    job_script_template_delete_file(\n    id_or_identifier: int | str = Path(),\n    file_name: str = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_EDIT,\n            ensure_email=True,\n        )\n    ),\n)\n

    Delete a file from a job script template by id or identifier.

    job_script_template_garbage_collector async 
    job_script_template_garbage_collector(\n    background_tasks: BackgroundTasks,\n    secure_services: SecureService = Depends(\n        secure_services(Permissions.JOB_TEMPLATES_EDIT)\n    ),\n)\n

    Delete all unused files from jobbergate templates on the file storage.

    job_script_template_get async 
    job_script_template_get(\n    id_or_identifier: int | str = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_VIEW, commit=False\n        )\n    ),\n)\n

    Get a job script template by id or identifier.

    job_script_template_get_file async 
    job_script_template_get_file(\n    id_or_identifier: int | str = Path(),\n    file_name: str = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_VIEW, commit=False\n        )\n    ),\n)\n

    Get a job script template file by id or identifier.

    Note

    See https://fastapi.tiangolo.com/advanced/custom-response/#streamingresponse

    job_script_template_get_list async 
    job_script_template_get_list(\n    list_params: ListParams = Depends(),\n    include_null_identifier: bool = Query(False),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_VIEW, commit=False\n        )\n    ),\n)\n

    Get a list of job script templates.

    job_script_template_update async 
    job_script_template_update(\n    update_request: JobTemplateUpdateRequest,\n    id_or_identifier: int | str = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_EDIT,\n            ensure_email=True,\n        )\n    ),\n)\n

    Update a job script template by id or identifier.

    job_script_template_upload_file async 
    job_script_template_upload_file(\n    id_or_identifier: int | str = Path(),\n    file_type: FileType = Path(),\n    upload_file: UploadFile = File(\n        ..., description=\"File to upload\"\n    ),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_EDIT,\n            ensure_email=True,\n        )\n    ),\n)\n

    Upload a file to a job script template by id or identifier.

    job_script_workflow_delete_file async 
    job_script_workflow_delete_file(\n    id_or_identifier: int | str = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_EDIT,\n            ensure_email=True,\n        )\n    ),\n)\n

    Delete a workflow file from a job script template by id or identifier.

    job_script_workflow_get_file async 
    job_script_workflow_get_file(\n    id_or_identifier: int | str = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_VIEW, commit=False\n        )\n    ),\n)\n

    Get a workflow file by id or identifier.

    Note

    See https://fastapi.tiangolo.com/advanced/custom-response/#streamingresponse

    job_script_workflow_upload_file async 
    job_script_workflow_upload_file(\n    id_or_identifier: int | str = Path(),\n    runtime_config: RunTimeConfig\n    | None = Body(\n        None,\n        description=\"Runtime configuration is optional when the workflow file already exists\",\n    ),\n    upload_file: UploadFile = File(\n        ..., description=\"File to upload\"\n    ),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_TEMPLATES_EDIT,\n            ensure_email=True,\n        )\n    ),\n)\n

    Upload a file to a job script workflow by id or identifier.

    "},{"location":"reference/api/#jobbergate_api.apps.job_script_templates.schemas","title":"schemas","text":"

    Provide schemas for the job script templates component.

    JobTemplateCreateRequest

    Bases: BaseModel

    Schema for the request to create a job template.

    JobTemplateDetailedView

    Bases: JobTemplateListView

    Schema for the request to an entry.

    Notice the files default to None, as they are not always requested, to differentiate between an empty list when they are requested, but no file is found.

    JobTemplateListView

    Bases: TableResource

    Schema for the response to get a list of entries.

    JobTemplateUpdateRequest

    Bases: BaseModel

    Schema for the request to update a job template.

    RunTimeConfig

    Bases: BaseModel

    Schema for the runtime config of a job template.

    Notice this includes user supplied variables, so it has no predefined field. It also loads the contend directly from the json at the request payload.

    __get_validators__ classmethod 
    __get_validators__()\n

    Get the validators.

    validate_to_json classmethod 
    validate_to_json(value)\n

    Validate the produced json.

    TemplateFileDetailedView

    Bases: BaseModel

    Schema for the response to get a template file.

    WorkflowFileDetailedView

    Bases: BaseModel

    Schema for the response to get a workflow file.

    "},{"location":"reference/api/#jobbergate_api.apps.job_script_templates.services","title":"services","text":"

    Services for the job_script_templates resource, including module specific business logic.

    JobScriptTemplateFileService

    Bases: FileService

    Provide an empty derived class of FileService.

    Although it doesn't do anything, it fixes errors with mypy: error: Value of type variable \"FileModel\" of \"FileService\" cannot be \"JobScriptTemplateFile\" error: Value of type variable \"FileModel\" of \"FileService\" cannot be \"WorkflowFile\"

    JobScriptTemplateService

    Bases: CrudService

    Provide a CrudService that overloads the list query builder and locator logic.

    build_list_query 
    build_list_query(\n    sort_ascending: bool = True,\n    search: str | None = None,\n    sort_field: str | None = None,\n    include_archived: bool = True,\n    include_files: bool = False,\n    include_parent: bool = False,\n    include_null_identifier: bool = True,\n    **additional_filters\n) -> Select\n

    List all job_script_templates.

    create async 
    create(**incoming_data) -> CrudModel\n

    Add a new row for the model to the database.

    locate_where_clause 
    locate_where_clause(id_or_identifier: Any) -> Any\n

    Locate an instance using the id or identifier field.

    update async 
    update(locator: Any, **incoming_data) -> CrudModel\n

    Update a row by locator with supplied data.

    validate_identifier 
    validate_identifier(identifier: str | None) -> None\n

    Validate that the identifier is not an empty string nor composed only by digits.

    Raise a ServiceError with status code 422 if the validation fails.

    Many of the job-script-template endpoints use the id or identifier interchangeably as a path parameter. With that, we need to ensure that the identifier is not a number, as that would be identified as id.

    "},{"location":"reference/api/#jobbergate_api.apps.job_scripts","title":"job_scripts","text":"

    Provide module for job_scripts.

    "},{"location":"reference/api/#jobbergate_api.apps.job_scripts.models","title":"models","text":"

    Database model for the JobScript resource.

    JobScript

    Bases: CrudMixin, Base

    Job script table definition.

    Notice all relationships are lazy=\"raise\" to prevent n+1 implicit queries. This means that the relationships must be explicitly eager loaded using helper functions in the class.

    Attributes:

    Name Type Description parent_template_id Mapped[int]

    The id of the parent template.

    See Mixin class definitions for other columns.

    include_files classmethod 
    include_files(query: Select) -> Select\n

    Include custom options on a query to eager load files.

    include_parent classmethod 
    include_parent(query: Select) -> Select\n

    Include custom options on a query to eager load parent data.

    sortable_fields classmethod 
    sortable_fields()\n

    Add parent_template_id as a sortable field.

    JobScriptFile

    Bases: FileMixin, Base

    Job script files table definition.

    Attributes:

    Name Type Description parent_template_id

    The id of the parent template.

    file_type Mapped[FileType]

    The type of the file.

    See Mixin class definitions for other columns

    "},{"location":"reference/api/#jobbergate_api.apps.job_scripts.routers","title":"routers","text":"

    Router for the Job Script Template resource.

    job_script_auto_clean_unused_entries 
    job_script_auto_clean_unused_entries(\n    background_tasks: BackgroundTasks,\n    secure_services: SecureService = Depends(\n        secure_services(Permissions.JOB_SCRIPTS_EDIT)\n    ),\n)\n

    Automatically clean unused job scripts depending on a threshold.

    job_script_create async 
    job_script_create(\n    create_request: JobScriptCreateRequest,\n    secure_services: SecureService = Depends(\n        secure_services(Permissions.JOB_SCRIPTS_EDIT)\n    ),\n)\n

    Create a stand alone job script.

    job_script_create_from_template async 
    job_script_create_from_template(\n    create_request: JobScriptCreateRequest,\n    render_request: RenderFromTemplateRequest,\n    id_or_identifier: int | str = Path(...),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SCRIPTS_EDIT, ensure_email=True\n        )\n    ),\n)\n

    Create a new job script from a job script template.

    job_script_delete async 
    job_script_delete(\n    id: int = Path(...),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SCRIPTS_EDIT, ensure_email=True\n        )\n    ),\n)\n

    Delete a job script template by id or identifier.

    job_script_delete_file async 
    job_script_delete_file(\n    id: int = Path(...),\n    file_name: str = Path(...),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SCRIPTS_EDIT, ensure_email=True\n        )\n    ),\n)\n

    Delete a file from a job script template by id or identifier.

    job_script_garbage_collector 
    job_script_garbage_collector(\n    background_tasks: BackgroundTasks,\n    secure_services: SecureService = Depends(\n        secure_services(Permissions.JOB_SCRIPTS_EDIT)\n    ),\n)\n

    Delete all unused files from job scripts on the file storage.

    job_script_get async 
    job_script_get(\n    id: int = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SCRIPTS_VIEW, commit=False\n        )\n    ),\n)\n

    Get a job script by id.

    job_script_get_file async 
    job_script_get_file(\n    id: int = Path(...),\n    file_name: str = Path(...),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SCRIPTS_VIEW, commit=False\n        )\n    ),\n)\n

    Get a job script file.

    Note

    See https://fastapi.tiangolo.com/advanced/custom-response/#streamingresponse

    job_script_get_list async 
    job_script_get_list(\n    list_params: ListParams = Depends(),\n    from_job_script_template_id: int\n    | None = Query(\n        None,\n        description=\"Filter job-scripts by the job-script-template-id they were created from.\",\n    ),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SCRIPTS_VIEW, commit=False\n        )\n    ),\n)\n

    Get a list of job scripts.

    job_script_update async 
    job_script_update(\n    update_params: JobScriptUpdateRequest,\n    id: int = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SCRIPTS_EDIT, ensure_email=True\n        )\n    ),\n)\n

    Update a job script template by id or identifier.

    job_script_upload_file async 
    job_script_upload_file(\n    id: int = Path(...),\n    file_type: FileType = Path(...),\n    upload_file: UploadFile = File(\n        ..., description=\"File to upload\"\n    ),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SCRIPTS_EDIT, ensure_email=True\n        )\n    ),\n)\n

    Upload a file to a job script.

    "},{"location":"reference/api/#jobbergate_api.apps.job_scripts.schemas","title":"schemas","text":"

    JobScript resource schema.

    JobScriptCreateRequest

    Bases: BaseModel

    Request model for creating JobScript instances.

    JobScriptDetailedView

    Bases: JobScriptListView

    Model to match database for the JobScript resource.

    JobScriptFileDetailedView

    Bases: BaseModel

    Model for the job_script_files field of the JobScript resource.

    JobScriptListView

    Bases: TableResource

    Model to match database for the JobScript resource.

    JobScriptUpdateRequest

    Bases: BaseModel

    Request model for updating JobScript instances.

    RenderFromTemplateRequest

    Bases: BaseModel

    Request model for creating a JobScript entry from a template.

    "},{"location":"reference/api/#jobbergate_api.apps.job_scripts.services","title":"services","text":"

    Services for the job_scripts resource, including module specific business logic.

    AutoCleanResponse

    Bases: NamedTuple

    Named tuple for the response of auto_clean_unused_job_scripts.

    JobScriptCrudService

    Bases: CrudService

    Provide an empty derived class of CrudService.

    Although it doesn't do anything, it fixes an error with mypy: error: Value of type variable \"CrudModel\" of \"CrudService\" cannot be \"JobScript\"

    auto_clean_unused_job_scripts async 
    auto_clean_unused_job_scripts() -> AutoCleanResponse\n

    Automatically clean unused job scripts depending on a threshold.

    Based on the last time each job script was updated or used to create a job submission, this will archived job scripts that were unarchived and delete jos script that were archived.

    delete async 
    delete(locator: Any) -> None\n

    Extend delete a row by locator.

    Orphaned job-scripts are now allowed on Jobbergate. However, the agent relies on them to submit jobs after requesting GET /agent/pending. This creates a race condition and errors occur when a job-script is deleted before the agent handles its submissions.

    To avoid this, they are marked as reject in this scenario.

    JobScriptFileService

    Bases: FileService

    Provide an empty derived class of FileService.

    Although it doesn't do anything, it fixes an error with mypy: error: Value of type variable \"FileModel\" of \"FileService\" cannot be \"JobScriptFile\"

    upsert async 
    upsert(\n    parent_id: int,\n    filename: str,\n    upload_content: str | bytes | UploadFile,\n    **upsert_kwargs\n) -> FileModel\n

    Upsert a file instance.

    validate_entrypoint_file async 
    validate_entrypoint_file(parent_id: int, filename: str)\n

    Validate that the entrypoint file is unique.

    "},{"location":"reference/api/#jobbergate_api.apps.job_scripts.tools","title":"tools","text":"

    Provide a convenience class for managing job-script files.

    inject_sbatch_params 
    inject_sbatch_params(\n    job_script_data_as_string: str, sbatch_params: list[str]\n) -> str\n

    Inject sbatch params into job script.

    Given the job script as job_script_data_as_string, inject the sbatch params in the correct location.

    "},{"location":"reference/api/#jobbergate_api.apps.job_submissions","title":"job_submissions","text":"

    Provide module for job_submissions.

    "},{"location":"reference/api/#jobbergate_api.apps.job_submissions.constants","title":"constants","text":"

    Describe constants for the job_submissions module.

    JobSubmissionStatus

    Bases: str, Enum

    Defines the set of possible statuses for a Job Submission.

    pretty_list classmethod 
    pretty_list()\n

    Return a comma-separated list of possible statuses.

    "},{"location":"reference/api/#jobbergate_api.apps.job_submissions.models","title":"models","text":"

    Database model for the JobSubmission resource.

    JobSubmission

    Bases: CrudMixin, Base

    Job submission table definition.

    Notice all relationships are lazy=\"raise\" to prevent n+1 implicit queries. This means that the relationships must be explicitly eager loaded using helper functions in the class.

    Attributes:

    Name Type Description job_script_id Mapped[int]

    Id number of the job scrip this submissions is based on.

    execution_directory Mapped[str]

    The directory where the job is executed.

    slurm_job_id Mapped[int]

    The id of the job in the slurm queue.

    client_id Mapped[str]

    The id of the custer this submission runs on.

    status Mapped[JobSubmissionStatus]

    The status of the job submission.

    report_message Mapped[str]

    The message returned by the job.

    execution_parameters Mapped[dict[str, Any]]

    The properties of the job.

    See Mixin class definitions for other columns

    include_files classmethod 
    include_files(query: Select) -> Select\n

    Include custom options on a query to eager load files.

    include_parent classmethod 
    include_parent(query: Select) -> Select\n

    Include custom options on a query to eager load parent data.

    searchable_fields classmethod 
    searchable_fields()\n

    Add client_id as a searchable field.

    sortable_fields classmethod 
    sortable_fields()\n

    Add additional sortable fields.

    "},{"location":"reference/api/#jobbergate_api.apps.job_submissions.properties_parser","title":"properties_parser","text":"

    Parser for Slurm REST API parameters from SBATCH parameters at the job script file.

    ArgumentParserCustomExit

    Bases: ArgumentParser

    Custom implementation of the built-in class for argument parsing.

    The sys.exit triggered by the original code is replaced by a ValueError, besides some friendly logging messages.

    exit 
    exit(status=0, message=None)\n

    Raise ValueError when parsing invalid parameters or if the type of their values is not correct.

    SbatchToSlurm dataclass

    Store the information for each parameter, including its name at Slurm API and SBATCH.

    Besides that, any extra argument this parameter needs when added to the parser. This information is used to build the jobscript/SBATCH parser and the two-way mapping between Slurm API and SBATCH names.

    build_mapping_sbatch_to_slurm 
    build_mapping_sbatch_to_slurm() -> bidict\n

    Create a mapper to translate in both ways between the names expected by Slurm REST API and SBATCH.

    build_parser 
    build_parser() -> ArgumentParser\n

    Build an ArgumentParser to handle all SBATCH parameters declared at sbatch_to_slurm.

    convert_sbatch_to_slurm_api 
    convert_sbatch_to_slurm_api(\n    input: Dict[str, Any]\n) -> Dict[str, Any]\n

    Take a dictionary containing key-value pairing of SBATCH parameter name space to Slurm API namespace.

    Notice the values should not be affected.

    Raise KeyError if any of the keys are unknown to the mapper.

    get_job_parameters 
    get_job_parameters(jobscript: str) -> Dict[str, Any]\n

    Parse all SBATCH parameters from a job script, map their names to Slurm API parameters.

    They are returned as a key-value pairing dictionary.

    get_job_properties_from_job_script 
    get_job_properties_from_job_script(\n    main_file_content: str, **kwargs\n) -> JobProperties\n

    Get the job properties for Slurm REST API from a job script file.

    Extra keyword arguments can be used to overwrite any parameter from the job script, like name or current_working_directory.

    jobscript_to_dict 
    jobscript_to_dict(\n    jobscript: str,\n) -> Dict[str, Union[str, bool]]\n

    Extract the SBATCH params from a given job script.

    It returns them in a dictionary for mapping the parameter names to their values.

    Raise ValueError if any of the parameters are unknown to the parser.

    "},{"location":"reference/api/#jobbergate_api.apps.job_submissions.routers","title":"routers","text":"

    Router for the JobSubmission resource.

    job_submission_agent_update async 
    job_submission_agent_update(\n    update_params: JobSubmissionAgentUpdateRequest,\n    id: int = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SUBMISSIONS_EDIT,\n            ensure_client_id=True,\n        )\n    ),\n)\n

    Update a job_submission with a new status.

    Make a put request to this endpoint with the new status to update a job_submission.

    job_submission_create async 
    job_submission_create(\n    create_request: JobSubmissionCreateRequest,\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SUBMISSIONS_EDIT,\n            ensure_email=True,\n        )\n    ),\n)\n

    Create a new job submission.

    Make a post request to this endpoint with the required values to create a new job submission.

    job_submission_delete async 
    job_submission_delete(\n    id: int = Path(\n        ...,\n        description=\"id of the job submission to delete\",\n    ),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SUBMISSIONS_EDIT,\n            ensure_email=True,\n        )\n    ),\n)\n

    Delete job_submission given its id.

    job_submission_get async 
    job_submission_get(\n    id: int = Path(...),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SUBMISSIONS_VIEW, commit=False\n        )\n    ),\n)\n

    Return the job_submission given it's id.

    job_submission_get_list async 
    job_submission_get_list(\n    list_params: ListParams = Depends(),\n    slurm_job_ids: str\n    | None = Query(\n        None,\n        description=\"Comma-separated list of slurm-job-ids to match active job_submissions\",\n    ),\n    submit_status: JobSubmissionStatus\n    | None = Query(\n        None,\n        description=\"Limit results to those with matching status\",\n    ),\n    from_job_script_id: int\n    | None = Query(\n        None,\n        description=\"Filter job-submissions by the job-script-id they were created from.\",\n    ),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SUBMISSIONS_VIEW, commit=False\n        )\n    ),\n)\n

    List job_submissions for the authenticated user.

    job_submission_update async 
    job_submission_update(\n    update_params: JobSubmissionUpdateRequest,\n    id: int = Path(),\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SUBMISSIONS_EDIT,\n            ensure_email=True,\n        )\n    ),\n)\n

    Update a job_submission given its id.

    job_submissions_agent_active async 
    job_submissions_agent_active(\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SUBMISSIONS_VIEW,\n            commit=False,\n            ensure_client_id=True,\n        )\n    )\n)\n

    Get a list of active job submissions for the cluster-agent.

    job_submissions_agent_pending async 
    job_submissions_agent_pending(\n    secure_services: SecureService = Depends(\n        secure_services(\n            Permissions.JOB_SUBMISSIONS_VIEW,\n            commit=False,\n            ensure_client_id=True,\n        )\n    )\n)\n

    Get a list of pending job submissions for the cluster-agent.

    "},{"location":"reference/api/#jobbergate_api.apps.job_submissions.schemas","title":"schemas","text":"

    JobSubmission resource schema.

    ActiveJobSubmission

    Bases: BaseModel

    Specialized model for the cluster-agent to pull an active job_submission.

    JobProperties

    Bases: BaseModel

    Specialized model for job properties.

    See more details at: https://slurm.schedmd.com/rest_api.html

    JobSubmissionAgentUpdateRequest

    Bases: BaseModel

    Request model for updating JobSubmission instances.

    JobSubmissionCreateRequest

    Bases: BaseModel

    Request model for creating JobSubmission instances.

    JobSubmissionDetailedView

    Bases: JobSubmissionListView

    Complete model to match the database for the JobSubmission resource.

    JobSubmissionListView

    Bases: TableResource

    Partial model to match the database for the JobSubmission resource.

    JobSubmissionUpdateRequest

    Bases: BaseModel

    Request model for updating JobSubmission instances.

    PendingJobSubmission

    Bases: BaseModel

    Specialized model for the cluster-agent to pull pending job_submissions.

    Model also includes data from its job_script and application sources.

    "},{"location":"reference/api/#jobbergate_api.apps.job_submissions.services","title":"services","text":"

    Services for the job_submissions resource, including module specific business logic.

    JobSubmissionService

    Bases: CrudService

    Provide a CrudService that overloads the list query builder.

    build_list_query 
    build_list_query(\n    sort_ascending: bool = True,\n    search: str | None = None,\n    sort_field: str | None = None,\n    include_archived: bool = True,\n    include_files: bool = False,\n    include_parent: bool = False,\n    filter_slurm_job_ids: list[int] | None = None,\n    **additional_filters\n) -> Select\n

    List all job_script_templates.

    "},{"location":"reference/api/#jobbergate_api.apps.models","title":"models","text":"

    Functionalities to be shared by all models.

    "},{"location":"reference/api/#jobbergate_api.apps.models.ArchiveMixin","title":"ArchiveMixin","text":"

    Add is_archived column to a table.

    Attributes:

    Name Type Description is_archived Mapped[bool]

    Specify is a row is considered archived, hidden it by default when listing rows.

    "},{"location":"reference/api/#jobbergate_api.apps.models.Base","title":"Base","text":"

    Bases: DeclarativeBase

    Base class for all models.

    References

    https://docs.sqlalchemy.org/en/20/orm/declarative_mixins.html

    "},{"location":"reference/api/#jobbergate_api.apps.models.CommonMixin","title":"CommonMixin","text":"

    Provide a dynamic table and helper methods for displaying instances.

    __str__ 
    __str__()\n

    Produce a pretty string representation of the class instance.

    __tablename__ classmethod 
    __tablename__() -> str\n

    Dynamically create table name based on the class name.

    "},{"location":"reference/api/#jobbergate_api.apps.models.CrudMixin","title":"CrudMixin","text":"

    Bases: CommonMixin, IdMixin, TimestampMixin, OwnerMixin, NameMixin, ArchiveMixin

    Add needed columns and declared attributes for all models that support a CrudService.

    include_files classmethod 
    include_files(query: Select) -> Select\n

    Include custom options on a query to eager load files.

    This should be overridden by derived classes.

    include_parent classmethod 
    include_parent(query: Select) -> Select\n

    Include custom options on a query to eager load parent data.

    This should be overridden by derived classes.

    searchable_fields classmethod 
    searchable_fields()\n

    Describe the fields that may be used in search queries.

    sortable_fields classmethod 
    sortable_fields()\n

    Describe the fields that may be used for sorting queries.

    "},{"location":"reference/api/#jobbergate_api.apps.models.FileMixin","title":"FileMixin","text":"

    Bases: CommonMixin, TimestampMixin

    Add needed columns and declared attributes for all models that support a FileService.

    Attributes:

    Name Type Description parent_id Mapped[int]

    The id of the parent row in another table. Note: Derived classes should override this attribute to make it a foreign key as well.

    description Mapped[int]

    The description of the job script template.

    file_key 
    file_key() -> str\n

    Dynamically define the s3 key for the file.

    "},{"location":"reference/api/#jobbergate_api.apps.models.IdMixin","title":"IdMixin","text":"

    Provide an id primary_key column.

    Attributes:

    Name Type Description id Mapped[int]

    The id of the job script template.

    "},{"location":"reference/api/#jobbergate_api.apps.models.NameMixin","title":"NameMixin","text":"

    Add name and description columns to a table.

    Attributes:

    Name Type Description name Mapped[str]

    The name of the job script template.

    description Mapped[str | None]

    The description of the job script template.

    "},{"location":"reference/api/#jobbergate_api.apps.models.OwnerMixin","title":"OwnerMixin","text":"

    Add an owner email columns to a table.

    Attributes:

    Name Type Description owner_email Mapped[str]

    The email of the owner of the job script template.

    "},{"location":"reference/api/#jobbergate_api.apps.models.TimestampMixin","title":"TimestampMixin","text":"

    Add timestamp columns to a table.

    Attributes:

    Name Type Description created_at Mapped[DateTime]

    The date and time when the job script template was created.

    updated_at Mapped[DateTime]

    The date and time when the job script template was updated.

    "},{"location":"reference/api/#jobbergate_api.apps.permissions","title":"permissions","text":"

    Provide a module that describes permissions in the API.

    "},{"location":"reference/api/#jobbergate_api.apps.permissions.Permissions","title":"Permissions","text":"

    Bases: str, Enum

    Describe the permissions that may be used for protecting Jobbergate routes.

    "},{"location":"reference/api/#jobbergate_api.apps.schemas","title":"schemas","text":"

    Define app-wide, reusable pydantic schemas.

    "},{"location":"reference/api/#jobbergate_api.apps.schemas.IgnoreLazyGetterDict","title":"IgnoreLazyGetterDict","text":"

    Bases: GetterDict

    A custom GetterDict to avoid triggering lazy-loads when accessing attributes.

    In this way, only explicitly joined relationships will be loaded and included in the response.

    References

    https://github.com/tiangolo/fastapi/discussions/5942

    __getitem__ 
    __getitem__(key: str) -> Any\n

    Customize getitem to avoid triggering lazy-loads when accessing attributes.

    get 
    get(key: Any, default: Any = None) -> Any\n

    Get an attribute value from the object, or return a default value if the attribute does not exist.

    "},{"location":"reference/api/#jobbergate_api.apps.schemas.ListParams","title":"ListParams","text":"

    Bases: BaseModel

    Describe the shared parameters for a list request.

    "},{"location":"reference/api/#jobbergate_api.apps.schemas.TableResource","title":"TableResource","text":"

    Bases: BaseModel

    Describes a base for table models that include basic, common info.

    "},{"location":"reference/api/#jobbergate_api.apps.services","title":"services","text":"

    Provide a generic services for CRUD and file operations in routers.

    "},{"location":"reference/api/#jobbergate_api.apps.services.BucketBoundService","title":"BucketBoundService","text":"

    Provide base class for services that bind to an s3 bucket.

    This class holds a reference to the bucket and provides methods to bind and unbind the bucket. It also keeps track of all instances of the service so that they can be iterated over.

    bucket property 
    bucket: Bucket\n

    Fetch the currently bound bucket.

    Raise an exception if the service is not bound to a bucket.

    __init__ 
    __init__()\n

    Initialize the service with a null bucket.

    bind_bucket 
    bind_bucket(bucket: Bucket)\n

    Bind the service to a bucket.

    bound_bucket 
    bound_bucket(bucket: Bucket)\n

    Provide a context within which the service is bound to a bucket.

    unbind_bucket 
    unbind_bucket()\n

    Unbind the service from a bucket.

    "},{"location":"reference/api/#jobbergate_api.apps.services.CrudModelProto","title":"CrudModelProto","text":"

    Bases: Protocol

    Provide a protocol for models that can be operated on by the CrudService.

    This protocol enables type hints for editors and type checking with mypy.

    These services would best be served by an intersection type so that the model_type is actually specified to inherit from both the mixins and the Base. This would allow static type checkers to recognize that all of the columns in a mixin are available and that the class can be instantiated in the create method. However, intersection types are not supported yet. For more information, see this discussion: https://github.com/python/typing/issues/213

    __init__ 
    __init__(**kwargs)\n

    Declare that the protocol can be instantiated.

    __tablename__ 
    __tablename__() -> str\n

    Declare that the protocol has a method to dynamically produce the table name.

    include_files classmethod 
    include_files(query: Select) -> Select\n

    Declare that the protocol has a method to include files in a query.

    include_parent classmethod 
    include_parent(query: Select) -> Select\n

    Declare that the protocol has a method to include details about the parent entry in a query.

    searchable_fields classmethod 
    searchable_fields() -> set[str]\n

    Declare that the protocol has searchable fields.

    sortable_fields classmethod 
    sortable_fields() -> set[str]\n

    Declare that the protocol has sortable fields.

    "},{"location":"reference/api/#jobbergate_api.apps.services.CrudService","title":"CrudService","text":"

    Bases: DatabaseBoundService, Generic[CrudModel]

    Provide a service that can perform various crud operations using a supplied ORM model type.

    name property 
    name\n

    Helper property to recover the name of the table.

    __init__ 
    __init__(model_type: type[CrudModel])\n

    Initialize the instance with an ORM model type.

    build_list_query 
    build_list_query(\n    sort_ascending: bool = True,\n    search: str | None = None,\n    sort_field: str | None = None,\n    include_archived: bool = True,\n    include_files: bool = False,\n    include_parent: bool = False,\n    **additional_filters\n) -> Select\n

    Build the query to list matching rows.

    Decomposed into a separate function so that deriving subclasses can add additional logic into the query.

    count async 
    count() -> int\n

    Count the number of rows in the table on the database.

    create async 
    create(**incoming_data) -> CrudModel\n

    Add a new row for the model to the database.

    delete async 
    delete(locator: Any) -> None\n

    Delete a row by locator.

    In almost all cases, the locator will just be an id value.

    ensure_attribute 
    ensure_attribute(instance: CrudModel, **attributes) -> None\n

    Ensure that a model instance has the specified values on key attributes.

    Raises HTTPException if the instance does not have the specified values.

    get async 
    get(\n    locator: Any,\n    include_files: bool = False,\n    include_parent: bool = False,\n    ensure_attributes: dict[str, Any] | None = None,\n) -> CrudModel\n

    Get a row by locator.

    In almost all cases, the locator will just be an id value.

    Key value pairs can be provided as ensure_attributes to assert that the key fields have the specified values. This is useful to assert email ownership of a row before modifying it, besides any other attribute.

    list async 
    list(**filter_kwargs) -> list[CrudModel]\n

    List all crud rows matching specified filters.

    For details on the supported filters, see the build_list_query() method.

    locate_where_clause 
    locate_where_clause(locator: Any) -> Any\n

    Provide the where clause expression to locate a row by locator.

    This method allows derived classes to locate by alternative identifiers, though locator is an id value in almost all cases. compound primary keys.

    paginated_list async 
    paginated_list(**filter_kwargs) -> Page[CrudModel]\n

    List all crud rows matching specified filters with pagination.

    For details on the supported filters, see the build_list_query() method.

    update async 
    update(locator: Any, **incoming_data) -> CrudModel\n

    Update a row by locator with supplied data.

    In almost all cases, the locator will just be an id value.

    "},{"location":"reference/api/#jobbergate_api.apps.services.DatabaseBoundService","title":"DatabaseBoundService","text":"

    Provide base class for services that bind to a database session.

    This class holds a reference to the session and provides methods to bind and unbind the session. It also keeps track of all instances of the service so that they can be iterated over.

    session property 
    session: AsyncSession\n

    Fetch the currently bound session.

    Raise an exception if the service is not bound to a session.

    __init__ 
    __init__()\n

    Instantiate the service with a null session.

    bind_session 
    bind_session(session: AsyncSession)\n

    Bind the service to a session.

    bound_session 
    bound_session(session: AsyncSession)\n

    Provide a context within which the service is bound to a session.

    unbind_session 
    unbind_session()\n

    Unbind the service from a session.

    "},{"location":"reference/api/#jobbergate_api.apps.services.FileModelProto","title":"FileModelProto","text":"

    Bases: Protocol

    Provide a protocol for models that can be operated on by the FileService.

    This protocol enables type hints for editors and type checking with mypy.

    These services would best be served by an intersection type so that the model_type is actually specified to inherit from both the mixins and the Base. This would allow static type checkers to recognize that all of the columns in a mixin are available and that the class can be instantiated in the create method. However, intersection types are not supported yet. For more information, see this discussion: https://github.com/python/typing/issues/213

    __init__ 
    __init__(**kwargs)\n

    Declare that the protocol can be instantiated.

    __tablename__ 
    __tablename__() -> str\n

    Declare that the protocol has a method to dynamically produce the table name.

    "},{"location":"reference/api/#jobbergate_api.apps.services.FileService","title":"FileService","text":"

    Bases: DatabaseBoundService, BucketBoundService, Generic[FileModel]

    Proide a service that can perform various file management operations using a supplied ORM model type.

    __init__ 
    __init__(model_type: type[FileModel])\n

    Initialize the instance with an ORM model type.

    delete async 
    delete(instance: FileModel) -> None\n

    Delete a file from s3 and from the corresponding table.

    find_children async 
    find_children(parent_id: int) -> list[FileModel]\n

    Find matching instances by parent_id.

    get async 
    get(parent_id: int, filename: str) -> FileModel\n

    Get a single instances by its parent id and filename (primary keys).

    Requires that one and only one result is found.

    get_file_content async 
    get_file_content(instance: FileModel) -> bytes\n

    Get the full contents for a file entry.

    render async 
    render(\n    instance: FileModel, parameters: dict[str, Any]\n) -> str\n

    Render the file using Jinja2.

    The parameters are passed to the template as the context, and two of them are supported: * Directly as the context, for instance, if the template contains {{ foo }}. * As a data key for backward compatibility, for instance, if the template contains {{ data.foo }}.

    stream_file_content async 
    stream_file_content(instance: FileModel) -> StreamingBody\n

    Stream the content of a file using a boto3 StreamingBody.

    The StreamingBody is an async generator that can be used for a StreamingResponse in a FastAPI app.

    upsert async 
    upsert(\n    parent_id: int,\n    filename: str,\n    upload_content: str | bytes | UploadFile,\n    **upsert_kwargs\n) -> FileModel\n

    Upsert a file instance.

    "},{"location":"reference/api/#jobbergate_api.apps.services.ServiceError","title":"ServiceError","text":"

    Bases: HTTPException

    Make HTTPException more friendly by changing the default behavior so that the first arg is a message.

    Also needed to play nice with py-buzz methods.

    __init__ 
    __init__(\n    message,\n    status_code=status.HTTP_400_BAD_REQUEST,\n    **kwargs\n)\n

    Instantiate the HTTPException super class by setting detail to the message provided.

    "},{"location":"reference/api/#jobbergate_api.config","title":"config","text":"

    Provide configuration settings for the app.

    Pull settings from environment variables or a .env file if available.

    "},{"location":"reference/api/#jobbergate_api.config.LogLevelEnum","title":"LogLevelEnum","text":"

    Bases: str, Enum

    Provide an enumeration class describing the available log levels.

    "},{"location":"reference/api/#jobbergate_api.config.Settings","title":"Settings","text":"

    Bases: BaseSettings

    Provide a pydantic BaseSettings model for the application settings.

    "},{"location":"reference/api/#jobbergate_api.config.Settings.remove_blank_env","title":"remove_blank_env","text":"
    remove_blank_env(values)\n

    Remove any settings from the environment that are blank strings.

    This allows the defaults to be set if docker-compose defaults a missing environment variable to a blank string.

    "},{"location":"reference/api/#jobbergate_api.config.check_none_or_all_keys_exist","title":"check_none_or_all_keys_exist","text":"
    check_none_or_all_keys_exist(\n    input_dict: dict, target_keys: set\n) -> bool\n

    Verify if none or all of the target keys exist in the input dictionary.

    "},{"location":"reference/api/#jobbergate_api.email_notification","title":"email_notification","text":"

    Email notification system for Jobbergate.

    "},{"location":"reference/api/#jobbergate_api.email_notification.EmailManager","title":"EmailManager dataclass","text":"

    Email manager.

    "},{"location":"reference/api/#jobbergate_api.email_notification.EmailManager.send_email","title":"send_email","text":"
    send_email(\n    to_emails: Union[str, List[str]],\n    subject: str,\n    skip_on_failure: bool = False,\n    **kwargs\n) -> None\n

    Send an email using this manager.

    "},{"location":"reference/api/#jobbergate_api.email_notification.EmailNotificationError","title":"EmailNotificationError","text":"

    Bases: Buzz

    Custom error to be raised for problems at the email notification system.

    "},{"location":"reference/api/#jobbergate_api.email_notification.notify_submission_rejected","title":"notify_submission_rejected","text":"
    notify_submission_rejected(\n    job_submission_id: Union[str, int],\n    report_message: str,\n    to_emails: Union[str, List[str]],\n) -> None\n

    Notify an email or a list of emails about a job submission that has been rejected.

    "},{"location":"reference/api/#jobbergate_api.main","title":"main","text":"

    Main file to startup the fastapi server.

    "},{"location":"reference/api/#jobbergate_api.main.health_check","title":"health_check async","text":"
    health_check()\n

    Provide a health-check endpoint for the app.

    "},{"location":"reference/api/#jobbergate_api.main.lifespan","title":"lifespan async","text":"
    lifespan(_: FastAPI)\n

    Provide a lifespan context for the app.

    Will set up logging and cleanup database engines when the app is shut down.

    This is the preferred method of handling lifespan events in FastAPI. For mor details, see: https://fastapi.tiangolo.com/advanced/events/

    "},{"location":"reference/api/#jobbergate_api.main.validation_exception_handler","title":"validation_exception_handler async","text":"
    validation_exception_handler(\n    request: Request, err: RequestValidationError\n)\n

    Handle exceptions from pydantic validators.

    "},{"location":"reference/api/#jobbergate_api.meta_mapper","title":"meta_mapper","text":"

    Provides a metadata-mapper for re-using descriptions and examples across many pydantic models.

    "},{"location":"reference/api/#jobbergate_api.meta_mapper.MetaField","title":"MetaField dataclass","text":"

    Provides a dataclass that describes the metadata that will be mapped for an individual field.

    "},{"location":"reference/api/#jobbergate_api.meta_mapper.MetaMapper","title":"MetaMapper","text":"

    Maps re-usable metadata for fields. Should be used with the schema_extra property of a Model's Config.

    Example::

    foo_meta = MetaMapper(\n    id=MetaField(\n        description=\"The unique identifier of this Foo\",\n        example=13,\n    ),\n    name=MetaField(\n        description=\"The name of this Foo\",\n        example=\"Bar\",\n    ),\n    is_active=MetaField(\n        description=\"Indicates if this Foo is active\",\n        example=True,\n    ),\n    created_at=MetaField(\n        description=\"The timestamp indicating when this Foo was created\",\n        example=\"2023-08-18T13:55:37.172285\",\n    ),\n)\n\n\nclass CreateFooRequest(BaseModel):\n    name: str\n    is_active: Optional[bool]\n\n    class Config:\n        schema_extra = foo_meta\n\n\nclass UpdateFooRequest(BaseModel):\n    name: Optional[str] = None\n    is_active: Optional[bool] = None\n\n    class Config:\n        schema_extra = foo_meta\n\n\nclass FooResponse(BaseModel):\n    id: int\n    name: str\n    is_active: bool\n    created_at: DateTime\n\n    class Config:\n        schema_extra = foo_meta\n

    Notice in this example that the fields may be required in some models and optional in others. Further, not all the fields are present in all the models. The MetaMapper allows the models to share field metadata and yet define the fields independently.

    "},{"location":"reference/api/#jobbergate_api.meta_mapper.MetaMapper.__call__","title":"__call__","text":"
    __call__(schema: Dict[str, Any], *_) -> None\n

    Map the MetaFields onto the metadata properties of a schema.

    Should be used in a pydantic Model's Config class.

    "},{"location":"reference/api/#jobbergate_api.meta_mapper.MetaMapper.__init__","title":"__init__","text":"
    __init__(**kwargs: MetaField)\n

    Map the kwargs into the field_dict.

    All kwargs should be MetaFields, but any object duck-typed to include all the attributes of a MetaField will be accepted.

    "},{"location":"reference/api/#jobbergate_api.safe_types","title":"safe_types","text":"

    Provide \"safe\" type annotatons to avoid issues with mypy and Fast api.

    Regarding the JobScript and JobSubmission type

    These are needed for the relationships in the models. This avoids issues with circular imports at runtime.

    Regarding the Bucket type

    This is necessary because the Bucket type isn't importable from the normal boto3 modules. Instead, it must be imported from the mypy typing plugin for boto3.

    The \"type\" must be bound to Any when not type checking because FastAPI does type inspection for its dependency injection system. Thus, there must be a type associated with Bucket even when not type checking.

    "},{"location":"reference/api/#jobbergate_api.security","title":"security","text":"

    Instantiates armasec resources for auth on api endpoints using project settings.

    Also provides a factory function for TokenSecurity to reduce boilerplate.

    "},{"location":"reference/api/#jobbergate_api.security.IdentityPayload","title":"IdentityPayload","text":"

    Bases: TokenPayload

    Provide an extension of TokenPayload that includes the user's identity.

    "},{"location":"reference/api/#jobbergate_api.security.IdentityPayload.extract_organization","title":"extract_organization","text":"
    extract_organization(values)\n

    Extract the organization_id from the organization payload.

    The payload is expected to look like: { ..., \"organization\": { \"adf99e01-5cd5-41ac-a1af-191381ad7780\": { ... } } }

    "},{"location":"reference/api/#jobbergate_api.security.get_domain_configs","title":"get_domain_configs","text":"
    get_domain_configs() -> list[DomainConfig]\n

    Return a list of DomainConfig objects based on the input variables for the Settings class.

    "},{"location":"reference/api/#jobbergate_api.security.lockdown_with_identity","title":"lockdown_with_identity","text":"
    lockdown_with_identity(\n    *scopes: str,\n    permission_mode: PermissionMode = PermissionMode.ALL,\n    ensure_email: bool = False,\n    ensure_organization: bool = False,\n    ensure_client_id: bool = False\n)\n

    Provide a wrapper to be used with dependency injection to extract identity on a secured route.

    "},{"location":"reference/api/#jobbergate_api.storage","title":"storage","text":"

    Provide functions to interact with persistent data storage.

    "},{"location":"reference/api/#jobbergate_api.storage.EngineFactory","title":"EngineFactory","text":"

    Provide a factory class that creates engines and keeps track of them in an engine mapping.

    This is used for multi-tenancy and database URL creation at request time.

    "},{"location":"reference/api/#jobbergate_api.storage.EngineFactory.__init__","title":"__init__","text":"
    __init__()\n

    Initialize the EngineFactory.

    "},{"location":"reference/api/#jobbergate_api.storage.EngineFactory.auto_session","title":"auto_session async","text":"
    auto_session(\n    override_db_name: str | None = None, commit: bool = True\n) -> typing.AsyncIterator[AsyncSession]\n

    Get an asynchronous database session.

    Gets a new session from the correct engine in the engine map.

    "},{"location":"reference/api/#jobbergate_api.storage.EngineFactory.cleanup","title":"cleanup async","text":"
    cleanup()\n

    Close all engines stored in the engine map and clears the engine_map.

    "},{"location":"reference/api/#jobbergate_api.storage.EngineFactory.get_engine","title":"get_engine","text":"
    get_engine(\n    override_db_name: str | None = None,\n) -> AsyncEngine\n

    Get a database engine.

    If the database url is already in the engine map, return the engine stored there. Otherwise, build a new one, store it, and return the new engine.

    "},{"location":"reference/api/#jobbergate_api.storage.SecureSession","title":"SecureSession dataclass","text":"

    Provide a container class for an IdentityPayload and AsyncSesson for the current request.

    "},{"location":"reference/api/#jobbergate_api.storage.build_db_url","title":"build_db_url","text":"
    build_db_url(\n    override_db_name: str | None = None,\n    force_test: bool = False,\n    asynchronous: bool = True,\n) -> str\n

    Build a database url based on settings.

    If force_test is set, build from the test database settings. If asynchronous is set, use asyncpg. If override_db_name replace the database name in the settings with the supplied value.

    "},{"location":"reference/api/#jobbergate_api.storage.handle_fk_error","title":"handle_fk_error","text":"
    handle_fk_error(\n    _: fastapi.Request,\n    err: asyncpg.exceptions.ForeignKeyViolationError,\n)\n

    Unpack metadata from a ForeignKeyViolationError and return a 409 response.

    "},{"location":"reference/api/#jobbergate_api.storage.render_sql","title":"render_sql","text":"
    render_sql(session: AsyncSession, query) -> str\n

    Render a sqlalchemy query into a string for debugging.

    "},{"location":"reference/api/#jobbergate_api.storage.search_clause","title":"search_clause","text":"
    search_clause(\n    search_terms: str, searchable_fields: set\n) -> ColumnElement[bool]\n

    Create search clause across searchable fields with search terms.

    Regarding the False first argument to or_(): The or_() function must have one fixed positional argument. See: https://docs.sqlalchemy.org/en/20/core/sqlelement.html#sqlalchemy.sql.expression.or_

    "},{"location":"reference/api/#jobbergate_api.storage.secure_session","title":"secure_session","text":"
    secure_session(\n    *scopes: str,\n    permission_mode: PermissionMode = PermissionMode.ALL,\n    commit: bool = True,\n    ensure_email: bool = False,\n    ensure_organization: bool = False,\n    ensure_client_id: bool = False\n)\n

    Provide an injectable for FastAPI that checks permissions and returns a database session for this request.

    This should be used for all secured routes that need access to the database. It will commit the transaction upon completion of the request. If an exception occurs, it will rollback the transaction. If multi-tenancy is enabled, it will retrieve a database session for the database associated with the client_id found in the requesting user's auth token.

    If testing mode is enabled, it will flush the session instead of committing changes to the database.

    Note that the session should NEVER be explicitly committed anywhere else in the source code.

    "},{"location":"reference/api/#jobbergate_api.storage.sort_clause","title":"sort_clause","text":"
    sort_clause(\n    sort_field: str,\n    sortable_fields: set,\n    sort_ascending: bool,\n) -> typing.Union[Mapped, UnaryExpression, Case]\n

    Create a sort clause given a sort field, the list of sortable fields, and a sort_ascending flag.

    "},{"location":"reference/api/#jobbergate_api.version","title":"version","text":"

    Provide the version of the package.

    "},{"location":"reference/api/#jobbergate_api.version.get_version","title":"get_version","text":"
    get_version() -> str\n

    Get the version from the metadata if available, otherwise from pyproject.toml.

    Returns \"unknown\" if both methods fail.

    "},{"location":"reference/api/#jobbergate_api.version.get_version_from_metadata","title":"get_version_from_metadata","text":"
    get_version_from_metadata() -> str\n

    Get the version from the metadata.

    This is the preferred method of getting the version, but only works if the package is properly installed in a Python environment.

    "},{"location":"reference/api/#jobbergate_api.version.get_version_from_poetry","title":"get_version_from_poetry","text":"
    get_version_from_poetry() -> str\n

    Get the version from pyproject.toml.

    This is a fallback method if the package is not installed, but just copied and accessed locally, like in a Docker image.

    "},{"location":"reference/cli/","title":"Jobbergate CLI Reference","text":""},{"location":"reference/cli/#jobbergate_cli","title":"jobbergate_cli","text":"

    Jobbergate command-line interface and app library

    "},{"location":"reference/cli/#jobbergate_cli.__getattr__","title":"__getattr__","text":"
    __getattr__(name: str)\n

    Overload module attribute lookup to warn if 'appform' is being imported because it is deprecated.

    "},{"location":"reference/cli/#jobbergate_cli.application_base","title":"application_base","text":"

    Provide a stub module to maintain compatibility with previous versions.

    Issue a deprecation warning when this module is imported from if JOBBERGATE_COMPATIBILITY_MODE is enabled.

    If JOBBERGATE_COMPATIBILITY_MODE is not enabled, raise an import error when this module is imported.

    "},{"location":"reference/cli/#jobbergate_cli.auth","title":"auth","text":"

    Utilities for handling auth in jobbergate-cli.

    "},{"location":"reference/cli/#jobbergate_cli.auth.clear_token_cache","title":"clear_token_cache","text":"
    clear_token_cache()\n

    Clears the token cache.

    "},{"location":"reference/cli/#jobbergate_cli.auth.fetch_auth_tokens","title":"fetch_auth_tokens","text":"
    fetch_auth_tokens(ctx: JobbergateContext) -> TokenSet\n

    Fetch an access token (and possibly a refresh token) from Auth0.

    Prints out a URL for the user to use to authenticate and polls the token endpoint to fetch it when the browser-based process finishes

    "},{"location":"reference/cli/#jobbergate_cli.auth.init_persona","title":"init_persona","text":"
    init_persona(\n    ctx: JobbergateContext,\n    token_set: Optional[TokenSet] = None,\n)\n

    Initializes the \"persona\" which contains the tokens and email address for a user.

    Retrieves the access token for the user from the cache.

    Token is retrieved from the cache, validated, and user email is extracted.

    If the access token is expired, a new one will be acquired via the cached refresh token (if there is one).

    Saves token_set to cache.

    Returns the persona.

    "},{"location":"reference/cli/#jobbergate_cli.auth.load_tokens_from_cache","title":"load_tokens_from_cache","text":"
    load_tokens_from_cache() -> TokenSet\n

    Loads an access token (and a refresh token if one exists) from the cache.

    "},{"location":"reference/cli/#jobbergate_cli.auth.open_on_browser","title":"open_on_browser","text":"
    open_on_browser(url: str) -> bool\n

    Open the url on the browser using webbrowser.

    "},{"location":"reference/cli/#jobbergate_cli.auth.refresh_access_token","title":"refresh_access_token","text":"
    refresh_access_token(\n    ctx: JobbergateContext, token_set: TokenSet\n)\n

    Attempt to fetch a new access token given a refresh token in a token_set.

    Sets the access token in-place.

    If refresh fails, notify the user that they need to log in again.

    "},{"location":"reference/cli/#jobbergate_cli.auth.save_tokens_to_cache","title":"save_tokens_to_cache","text":"
    save_tokens_to_cache(token_set: TokenSet)\n

    Saves tokens from a token_set to the cache.

    "},{"location":"reference/cli/#jobbergate_cli.auth.show_login_message","title":"show_login_message","text":"
    show_login_message(verification_uri: str)\n

    Show a message to the user with a link to the auth provider to login.

    "},{"location":"reference/cli/#jobbergate_cli.auth.validate_token_and_extract_identity","title":"validate_token_and_extract_identity","text":"
    validate_token_and_extract_identity(\n    token_set: TokenSet,\n) -> IdentityData\n

    Validate the access_token from a TokenSet and extract the user's identity data.

    Validations
    • Checks if access_token is not empty.
    • Checks timestamp on the access token.
    • Checks that the client_id is present
    • Checks that email is present

    Reports an error in the logs and to the user if there is an issue with the access_token.

    "},{"location":"reference/cli/#jobbergate_cli.compat","title":"compat","text":"

    Provide compatibility to the previous version of Jobbergate CLI for users who have automation or are familiar with the old commands

    "},{"location":"reference/cli/#jobbergate_cli.compat.add_legacy_compatible_commands","title":"add_legacy_compatible_commands","text":"
    add_legacy_compatible_commands(app: typer.Typer)\n

    Add commands from the restructured CLI under the previous names for the commands to the root typer app.

    "},{"location":"reference/cli/#jobbergate_cli.config","title":"config","text":"

    Configuration file, sets all the necessary environment variables. Can load configuration from a dotenv file if supplied.

    "},{"location":"reference/cli/#jobbergate_cli.config.Settings","title":"Settings","text":"

    Bases: BaseSettings

    Provide a pydantic settings model to hold configuration values loaded from the environment.

    "},{"location":"reference/cli/#jobbergate_cli.config.Settings.Config","title":"Config","text":"

    Customize behavior of the Settings class. Especially, enable the use of dotenv to load settings from a .env file instead of the environment.

    "},{"location":"reference/cli/#jobbergate_cli.config.Settings.compute_extra_settings","title":"compute_extra_settings","text":"
    compute_extra_settings(values)\n

    Compute settings values that are based on other settings values.

    "},{"location":"reference/cli/#jobbergate_cli.config.build_settings","title":"build_settings","text":"
    build_settings(*args, **kwargs)\n

    Return a Setting object and handle ValidationError with a message to the user.

    "},{"location":"reference/cli/#jobbergate_cli.constants","title":"constants","text":"

    Provide constants that may be used throughout the CLI modules.

    "},{"location":"reference/cli/#jobbergate_cli.constants.FileType","title":"FileType","text":"

    Bases: str, Enum

    File type enum.

    "},{"location":"reference/cli/#jobbergate_cli.constants.SortOrder","title":"SortOrder","text":"

    Bases: str, Enum

    Enum descring the type of sort orders that are available for list commands.

    "},{"location":"reference/cli/#jobbergate_cli.exceptions","title":"exceptions","text":"

    Provide exceptions and custom handlers for the CLI.

    "},{"location":"reference/cli/#jobbergate_cli.exceptions.Abort","title":"Abort","text":"

    Bases: Buzz

    A special exception used to abort the Jobbergate CLI.

    Collects information provided for use in the handle_abort context manager.

    "},{"location":"reference/cli/#jobbergate_cli.exceptions.Abort.__init__","title":"__init__","text":"
    __init__(\n    message,\n    *args,\n    subject=None,\n    support=False,\n    log_message=None,\n    sentry_context=None,\n    original_error=None,\n    warn_only=False,\n    **kwargs\n)\n

    Initialize the Abort errror.

    "},{"location":"reference/cli/#jobbergate_cli.exceptions.JobbergateCliError","title":"JobbergateCliError","text":"

    Bases: Buzz

    A generic exception base class to use in Jobbergate CLI

    "},{"location":"reference/cli/#jobbergate_cli.exceptions.handle_abort","title":"handle_abort","text":"
    handle_abort(func)\n

    Apply a decorator to gracefully handle any Abort errors that happen within the context.

    Will log the error, dispatch it to Sentry, show a helpful message to the user about the error, and exit.

    "},{"location":"reference/cli/#jobbergate_cli.jobberappslib","title":"jobberappslib","text":"

    Provide a stub module to maintain compatibility with previous versions.

    Issue a deprecation warning when this module is imported from if JOBBERGATE_COMPATIBILITY_MODE is enabled.

    If JOBBERGATE_COMPATIBILITY_MODE is not enabled, raise an import error when this module is imported.

    "},{"location":"reference/cli/#jobbergate_cli.logging","title":"logging","text":"

    Provide initializers for logging.

    "},{"location":"reference/cli/#jobbergate_cli.logging.init_logs","title":"init_logs","text":"
    init_logs(verbose=False)\n

    Initialize logging.

    If JOBBERGATE_LOG_PATH is set in the config, add a rotatating file log handler. Logs will be retained for 1 week.

    If verbose is supplied, add a stdout handler at the DEBUG level.

    "},{"location":"reference/cli/#jobbergate_cli.logging.init_sentry","title":"init_sentry","text":"
    init_sentry()\n

    Initialize Sentry if the SENTRY_DSN environment variable is present.

    "},{"location":"reference/cli/#jobbergate_cli.main","title":"main","text":"

    Provide main entry point for the Jobbergate CLI App.

    "},{"location":"reference/cli/#jobbergate_cli.main.login","title":"login","text":"
    login(ctx: typer.Context)\n

    Log in to the jobbergate-cli by storing the supplied token argument in the cache.

    "},{"location":"reference/cli/#jobbergate_cli.main.logout","title":"logout","text":"
    logout()\n

    Logs out of the jobbergate-cli. Clears the saved user credentials.

    "},{"location":"reference/cli/#jobbergate_cli.main.main","title":"main","text":"
    main(\n    ctx: typer.Context,\n    verbose: bool = typer.Option(\n        False, help=\"Enable verbose logging to the terminal\"\n    ),\n    full: bool = typer.Option(\n        False, help=\"Print all fields from CRUD commands\"\n    ),\n    raw: bool = typer.Option(\n        False,\n        help=\"Print output from CRUD commands as raw json\",\n    ),\n    version: bool = typer.Option(\n        False,\n        help=\"Print the version of jobbergate-cli and exit\",\n    ),\n    ignore_extra_args: str = typer.Option(\n        None,\n        \"--username\",\n        \"-u\",\n        \"--password\",\n        \"-p\",\n        hidden=True,\n        help=\"Ignore extra arguments passed to the command for backward compatibility with the legacy app.\",\n    ),\n)\n

    Welcome to the Jobbergate CLI!

    More information can be shown for each command listed below by running it with the --help option.

    "},{"location":"reference/cli/#jobbergate_cli.main.show_token","title":"show_token","text":"
    show_token(\n    plain: bool = typer.Option(\n        False, help=\"Show the token in plain text.\"\n    ),\n    refresh: bool = typer.Option(\n        False,\n        help=\"Show the refresh token instead of the access token.\",\n    ),\n    show_prefix: bool = typer.Option(\n        False,\n        \"--prefix\",\n        help=\"Include the 'Bearer' prefix in the output.\",\n    ),\n    show_header: bool = typer.Option(\n        False,\n        \"--header\",\n        help=\"Show the token as it would appear in a request header.\",\n    ),\n    decode: bool = typer.Option(\n        False,\n        \"--decode\",\n        help=\"Show the content of the decoded access token.\",\n    ),\n)\n

    Show the token for the logged in user.

    Token output is automatically copied to your clipboard.

    "},{"location":"reference/cli/#jobbergate_cli.render","title":"render","text":"

    Provide helpers to render output for users.

    "},{"location":"reference/cli/#jobbergate_cli.render.StyleMapper","title":"StyleMapper","text":"

    Provide a mapper that can set rich styles for rendered output of data tables and dicts.

    The subapps have list endpoints that return sets of values. These are rendered as tables in the output. The StyleMapper class provides a way to simply define styles that should be applied to the columns of the table.

    Example:

    The following code will print a table where the columns are colored according to the style_mapper

    .. code-block: python

    style_mapper = StyleMapper( a=\"bold green\", b=\"red\", c=\"blue\", ) envelope = dict( results=[ dict(a=1, b=2, c=3), dict(a=4, b=5, c=6), dict(a=7, b=8, c=9), ], pagination=dict(total=3) ) render_list_results(jb_ctx, envelope, style_mapper)

    "},{"location":"reference/cli/#jobbergate_cli.render.StyleMapper.__init__","title":"__init__","text":"
    __init__(**colors: str)\n

    Initialize the StyleMapper.

    "},{"location":"reference/cli/#jobbergate_cli.render.StyleMapper.map_style","title":"map_style","text":"
    map_style(column: str) -> Dict[str, Any]\n

    Map a column name from the table to display to the style that should be used to render it.

    "},{"location":"reference/cli/#jobbergate_cli.render.render_dict","title":"render_dict","text":"
    render_dict(\n    data: Dict[str, Any],\n    title: str = \"Data\",\n    hidden_fields: Optional[List[str]] = None,\n)\n

    Render a dictionary in a rich Table That shows the key and value of each item.

    :param: data: The dictionary to render :param: title: The title header to include above the Table output :param: hidden_fields: Keys that should be hidden in the Table output

    "},{"location":"reference/cli/#jobbergate_cli.render.render_json","title":"render_json","text":"
    render_json(data: Any)\n

    Print nicely formatted representation of a JSON serializable python primitive.

    "},{"location":"reference/cli/#jobbergate_cli.render.render_list_results","title":"render_list_results","text":"
    render_list_results(\n    ctx: JobbergateContext,\n    envelope: ListResponseEnvelope,\n    style_mapper: Optional[StyleMapper] = None,\n    hidden_fields: Optional[List[str]] = None,\n    title: str = \"Results List\",\n)\n

    Render a list of result data items in a rich Table.

    :param: ctx: The JobbergateContext. This is needed to detect if full or raw output is needed :param: envelope: A ListResponseEnvelope containing the data items :param: style_mapper: The style mapper that should be used to apply styles to the columns of the table :param: hidden_fields: Columns that should (if not using full mode) be hidden in the Table output :param: title: The title header to include above the Table output

    "},{"location":"reference/cli/#jobbergate_cli.render.render_single_result","title":"render_single_result","text":"
    render_single_result(\n    ctx: JobbergateContext,\n    result: Union[Dict[str, Any], pydantic.BaseModel],\n    hidden_fields: Optional[List[str]] = None,\n    title: str = \"Result\",\n)\n

    Render a single data item in a rich ``Table.

    :param: ctx: The JobbergateContext. This is needed to detect if full` orrawoutput is needed :param: result: The data item to display. May be a dict or a pydantic model. :param: hidden_fields: Rows that should (if not usingfullmode) be hidden in theTableoutput :param: title: The title header to include above theTale`` output

    "},{"location":"reference/cli/#jobbergate_cli.render.terminal_message","title":"terminal_message","text":"
    terminal_message(\n    message,\n    subject=None,\n    color=\"green\",\n    footer=None,\n    indent=True,\n)\n

    Print a nicely formatted message as output to the user using a rich Panel.

    :param: message: The message to print out :param: subject: An optional subject line to add in the header of the Panel :param: color: An optional color to style the subject header with :param: footer: An optional message to display in the footer of the Panel :param: indent: Adds padding to the left of the message

    "},{"location":"reference/cli/#jobbergate_cli.requests","title":"requests","text":"

    Provide utilities for making requests against the Jobbergate API.

    "},{"location":"reference/cli/#jobbergate_cli.requests.make_request","title":"make_request","text":"
    make_request(\n    client: httpx.Client,\n    url_path: str,\n    method: str,\n    *,\n    expected_status: Optional[int] = None,\n    expect_response: bool = True,\n    abort_message: str = \"There was an error communicating with the API\",\n    abort_subject: str = \"REQUEST FAILED\",\n    support: bool = True,\n    response_model_cls: Optional[\n        Type[ResponseModel]\n    ] = None,\n    request_model: Optional[pydantic.BaseModel] = None,\n    save_to_file: Optional[Path] = None,\n    **request_kwargs: Any\n) -> Union[ResponseModel, Dict, int]\n

    Make a request against the Jobbergate API.

    :param: client: The Httpx client to use for the request :param: url_path: The path to add to the base url of the client where the request should be sent :param: method: The REST method to use for the request (GET, PUT, UPDATE, POST, DELETE, etc) :param: expected_status: The status code to expect on the response. If it is not received, raise an Abort :param: expect_response: Indicates if response data (JSON) is expected from the API endpoint :param: abort_message: The message to show the user if there is a problem and the app must be aborted :param: abort_subject: The subject to use in Abort output to the user :param: support: If true, add a message to the output instructing the user to seek help :param: response_model_cls: If supplied, serialize the response data into this Pydantic model class :param: request_model: Use a pydantic model instance as the data body for the request :param: request_kwargs: Any additional keyword arguments that need to be passed on to the client

    "},{"location":"reference/cli/#jobbergate_cli.schemas","title":"schemas","text":"

    Provide Pydantic models for various data items.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.ApplicationResponse","title":"ApplicationResponse","text":"

    Bases: BaseModel

    Describes the format of data for applications retrieved from the Jobbergate API endpoints.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.ClusterCacheData","title":"ClusterCacheData","text":"

    Bases: BaseModel

    Describes the format of data stored in the clusters cache file.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.DeviceCodeData","title":"DeviceCodeData","text":"

    Bases: BaseModel

    A model representing the data that is returned from the OIDC provider's device code endpoint.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.IdentityData","title":"IdentityData","text":"

    Bases: BaseModel

    A model representing the identifying data for a user from an auth token.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobScriptCreateRequest","title":"JobScriptCreateRequest","text":"

    Bases: BaseModel

    Request model for creating JobScript instances.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobScriptFiles","title":"JobScriptFiles","text":"

    Bases: BaseModel

    Model containing job-script files.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobScriptRenderRequestData","title":"JobScriptRenderRequestData","text":"

    Bases: BaseModel

    Describes the data that will be sent to the create endpoint of the Jobbergate API for job scripts.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobScriptResponse","title":"JobScriptResponse","text":"

    Bases: BaseModel

    Describes the format of data for job_scripts retrieved from the Jobbergate API endpoints.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobScriptResponse.null_files","title":"null_files","text":"
    null_files(value)\n

    Remap a None value in files to an empty list.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobSubmissionCreateRequestData","title":"JobSubmissionCreateRequestData","text":"

    Bases: BaseModel

    Describes the data that will be sent to the create endpoint of the Jobbergate API for job submissions.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobSubmissionResponse","title":"JobSubmissionResponse","text":"

    Bases: BaseModel

    Describes the format of data for job_submissions retrieved from the Jobbergate API endpoints.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobbergateApplicationConfig","title":"JobbergateApplicationConfig","text":"

    Bases: BaseModel

    A data object describing the config data needed to instantiate a JobbergateApplication class.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobbergateConfig","title":"JobbergateConfig","text":"

    Bases: BaseModel

    A data object describing the config values needed in the \"jobbergate_config\" section of the JobbergateApplicationConfig model.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobbergateConfig.compute_extra_settings","title":"compute_extra_settings","text":"
    compute_extra_settings(values)\n

    Compute missing values and extra operations to enhance the user experience and backward compatibility.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.JobbergateContext","title":"JobbergateContext","text":"

    Bases: BaseModel

    A data object describing context passed from the main entry point.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.ListResponseEnvelope","title":"ListResponseEnvelope","text":"

    Bases: BaseModel

    A model describing the structure of response envelopes from \"list\" endpoints.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.Persona","title":"Persona","text":"

    Bases: BaseModel

    A model representing a pairing of a TokenSet and user email. This is a convenience to combine all of the identifying data and credentials for a given user.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.RenderFromTemplateRequest","title":"RenderFromTemplateRequest","text":"

    Bases: BaseModel

    Request model for creating a JobScript entry from a template.

    "},{"location":"reference/cli/#jobbergate_cli.schemas.TokenSet","title":"TokenSet","text":"

    Bases: BaseModel

    A model representing a pairing of access and refresh tokens

    "},{"location":"reference/cli/#jobbergate_cli.subapps","title":"subapps","text":"

    Subapps that are added to the base Typer application.

    "},{"location":"reference/cli/#jobbergate_cli.subapps.applications","title":"applications","text":"

    Provide a sub-app for interacting with Applications data.

    "},{"location":"reference/cli/#jobbergate_cli.subapps.applications.app","title":"app","text":"

    Provide a typer app that can interact with Application data in a cruddy manner.

    create 
    create(\n    ctx: typer.Context,\n    name: str = typer.Option(\n        ...,\n        \"--name\",\n        \"-n\",\n        help=\"The name of the application to create\",\n    ),\n    identifier: Optional[str] = typer.Option(\n        None,\n        help=f\"The human-friendly identifier of the application. {IDENTIFIER_NOTE}\",\n    ),\n    application_path: pathlib.Path = typer.Option(\n        ...,\n        \"--application-path\",\n        \"-a\",\n        help=\"The path to the directory where the application files are located\",\n    ),\n    application_desc: Optional[str] = typer.Option(\n        None,\n        help=\"A helpful description of the application\",\n    ),\n)\n

    Create a new application.

    delete 
    delete(\n    ctx: typer.Context,\n    id: Optional[int] = typer.Option(\n        None,\n        \"--id\",\n        \"-i\",\n        help=f\"The specific id of the application to delete. {ID_NOTE}\",\n    ),\n    identifier: Optional[str] = typer.Option(\n        None,\n        help=f\"The human-friendly identifier of the application to update. {IDENTIFIER_NOTE}\",\n    ),\n)\n

    Delete an existing application.

    download_files 
    download_files(\n    ctx: typer.Context,\n    id: Optional[int] = typer.Option(\n        None,\n        help=f\"The specific id of the application. {ID_NOTE}\",\n    ),\n    identifier: Optional[str] = typer.Option(\n        None,\n        help=f\"The human-friendly identifier of the application. {IDENTIFIER_NOTE}\",\n    ),\n)\n

    Download the files from an application to the current working directory.

    get_one 
    get_one(\n    ctx: typer.Context,\n    id: Optional[int] = typer.Option(\n        None,\n        \"--id\",\n        \"-i\",\n        help=f\"The specific id of the application. {ID_NOTE}\",\n    ),\n    identifier: Optional[str] = typer.Option(\n        None,\n        help=f\"The human-friendly identifier of the application. {IDENTIFIER_NOTE}\",\n    ),\n)\n

    Get a single application by id or identifier

    list_all 
    list_all(\n    ctx: typer.Context,\n    show_all: bool = typer.Option(\n        False,\n        \"--all\",\n        help=\"Show all applications, even the ones without identifier\",\n    ),\n    user_only: bool = typer.Option(\n        False,\n        \"--user\",\n        help=\"Show only applications owned by the current user\",\n    ),\n    search: Optional[str] = typer.Option(\n        None, help=\"Apply a search term to results\"\n    ),\n    sort_order: SortOrder = typer.Option(\n        SortOrder.UNSORTED, help=\"Specify sort order\"\n    ),\n    sort_field: Optional[str] = typer.Option(\n        None,\n        help=\"The field by which results should be sorted\",\n    ),\n)\n

    Show available applications

    update 
    update(\n    ctx: typer.Context,\n    id: Optional[int] = typer.Option(\n        None,\n        \"--id\",\n        \"-i\",\n        help=f\"The specific id of the application to update. {ID_NOTE}\",\n    ),\n    identifier: Optional[str] = typer.Option(\n        None,\n        help=f\"The human-friendly identifier of the application to update. {IDENTIFIER_NOTE}\",\n    ),\n    application_path: Optional[pathlib.Path] = typer.Option(\n        None,\n        \"--application-path\",\n        \"-a\",\n        help=\"The path to the directory where the application files are located\",\n    ),\n    update_identifier: Optional[str] = typer.Option(\n        None,\n        help=\"Optional new application identifier to be set\",\n    ),\n    application_desc: Optional[str] = typer.Option(\n        None,\n        help=\"Optional new application description to be set\",\n    ),\n    application_name: Optional[str] = typer.Option(\n        None, help=\"Optional new application name to be set\"\n    ),\n)\n

    Update an existing application.

    "},{"location":"reference/cli/#jobbergate_cli.subapps.applications.application_base","title":"application_base","text":"

    ApplicationBase.

    JobbergateApplicationBase

    JobbergateApplicationBase.

    __init__ 
    __init__(jobbergate_yaml: Dict[str, Any])\n

    Initialize class attributes.

    find_templates staticmethod 
    find_templates(\n    application_path: pathlib.Path,\n) -> List[pathlib.Path]\n

    Finds templates a given application path.

    mainflow 
    mainflow(data: Dict[str, Any])\n

    Implements the main question asking workflow.

    "},{"location":"reference/cli/#jobbergate_cli.subapps.applications.application_helpers","title":"application_helpers","text":"

    Helper functions that may be used inside of Jobbergate applications.

    get_file_list 
    get_file_list(path=None, search_term='*.*')\n

    Return a list of input files in a directory that match a search term.

    Ignore casing when comparing against the search term.

    Default to searching for all files in the current directory.

    get_running_jobs 
    get_running_jobs(user_only=True)\n

    Return a list of the user's currently running jobs, as given by SLURM's squeue command.

    The format returned is: [job ID, 8 chars] [job name]

    "},{"location":"reference/cli/#jobbergate_cli.subapps.applications.questions","title":"questions","text":"

    Abstraction layer for questions. Each class represents different question types.

    The questions describe literal questions that are asked of the user in an interactive mode via the inquirer package.

    Questions will be skipped and use the default value if the ignore property resolves to True.

    Questions will also resolve to their default values if running in \"fast mode\".

    BooleanList

    Bases: Confirm

    Asks a confirmation question that is followed up by a certain question list when true and a different list if false.

    __init__ 
    __init__(\n    variablename: str,\n    message: str,\n    whentrue=None,\n    whenfalse=None,\n    **kwargs\n)\n

    Initialize the Checkbox question.

    :param: variablename: The key in the config dictionary that this question will set :param: message: The message to show the user that describes what the question is gathering :param whentrue: List of questions to ask if user answers 'true' on this question :param whentrue: List of questions to show if user answers 'false' on this question

    ignore_child 
    ignore_child(\n    child: QuestionBase, answers: Dict[str, Any]\n) -> bool\n

    Dynamically check if a child question should be ignored based on the questions that have already been answered.

    :param: child: The child question that might be ignored :param: answers: Answer values to previously asked questions

    make_ignore_partial 
    make_ignore_partial(\n    child: QuestionBase,\n) -> Callable[[Dict[str, Any]], bool]\n

    Build a partial method for checking if a child should be ignored.

    This method just makes the code more readable so that a non-descriptive lambda does not need to be used inline.

    make_prompts 
    make_prompts(**override_kwargs)\n

    Create inquirer prompts from this instance of BooleanList and for all its child questions.

    :param: override_kwargs: A collection of keyword arguments to override in the base make_prompts method

    Checkbox

    Bases: QuestionBase

    Gives the user a list to choose multiple entries from.

    __init__ 
    __init__(\n    variablename: str, message: str, choices: list, **kwargs\n)\n

    Initialize the Checkbox question.

    :param: variablename: The key in the config dictionary that this question will set :param: message: The message to show the user that describes what the question is gathering :param: choices: A list of the possible values from which the Question will allow the user to select many

    Confirm

    Bases: QuestionBase

    Asks a question with a boolean answer (true/false).

    __init__ 
    __init__(variablename: str, message: str, **kwargs)\n

    Initialize the Confirm question.

    :param: variablename: The key in the config dictionary that this question will set :param: message: The message to show the user that describes what the question is gathering

    Const

    Bases: Text

    Sets the variable to the default value. Doesn't show anything.

    __init__ 
    __init__(variablename: str, **kwargs)\n

    Initialize the Const \"question\".

    :param: variablename: The key in the config dictionary that this question will set

    make_prompts 
    make_prompts()\n

    Create inquirer prompts from this instance of Const.

    Directory

    Bases: QuestionBase

    Asks for a directory name. If exists is True it checks if path exists and is a directory.

    :param exists: Checks if given directory exists

    __init__ 
    __init__(\n    variablename: str,\n    message: str,\n    exists: Optional[bool] = None,\n    **kwargs\n)\n

    Initialize the Directory question.

    :param: variablename: The key in the config dictionary that this question will set :param: message: The message to show the user that describes what the question is gathering :param: exists: If True, ensure that the directory exists on the system

    File

    Bases: QuestionBase

    Asks for a file name.

    __init__ 
    __init__(\n    variablename: str,\n    message: str,\n    exists: Optional[bool] = None,\n    **kwargs\n)\n

    Initialize the File question.

    :param: variablename: The key in the config dictionary that this question will set :param: message: The message to show the user that describes what the question is gathering :param: exists: If True, ensure that the file path exists on the system

    Integer

    Bases: QuestionBase

    Asks for an integer value. Could have min and/or max constrains.

    __init__ 
    __init__(\n    variablename: str,\n    message: str,\n    minval: Optional[int] = None,\n    maxval: Optional[int] = None,\n    **kwargs\n)\n

    Initialize the Integer question.

    :param: variablename: The key in the config dictionary that this question will set :param: message: The message to show the user that describes what the question is gathering :param: minval: The minimum value the integer may be set to. If not specified, use negative infinity. :param: minval: The maximum value the integer may be set to. If not specified, use infinity.

    List

    Bases: QuestionBase

    Gives the user a list to choose one from.

    __init__ 
    __init__(\n    variablename: str, message: str, choices: list, **kwargs\n)\n

    Initialize the List question.

    :param: variablename: The key in the config dictionary that this question will set :param: message: The message to show the user that describes what the question is gathering :param: choices: A list of the possible values from which the Question will allow the user to select one

    QuestionBase

    Baseclass for questions.

    All questions have variablename, message and an optional default.

    __init__ 
    __init__(\n    variablename: str,\n    message: str,\n    ignore: bool = False,\n    default: Optional[Any] = None,\n    inquirer_type: Type[TInquirerType] = inquirer.Text,\n)\n

    Initialize the Question.

    :param: variablename: The key in the config dictionary that this question will set :param: message: The message to show the user that describes what the question is gathering :param: ignore: If true, do not ask the question and just use the default value instead :param: default: The default value for the variablename in the answers dict :param: inquirer_type: The inquirer question type that this QuestionBase wraps

    make_prompts 
    make_prompts(**override_kwargs)\n

    Create inquirer prompts from this instance of QuestionBase.

    :param: override_kwargs: A collection of keyword arguments to override in intializing the inquirer question

    Text

    Bases: QuestionBase

    Asks for a text value.

    gather_param_values 
    gather_param_values(\n    application: JobbergateApplicationBase,\n    supplied_params: Optional[Dict[str, Any]] = None,\n    fast_mode: bool = False,\n) -> Dict[str, Any]\n

    Gather the parameter values by executing the application methods.

    Prompt users for answers or use defaults as needed.

    :param: application: The application instance to pull questions from :param: supplied_params: Pre-supplied parameters. Any questions where the variablename matches a pre-supplied key in the dict at the start of execution will be skipped. :param: fast_mode: Do not ask the user questions. Just use the supplied params and defaults. :returns: A dict of the gathered parameter values

    "},{"location":"reference/cli/#jobbergate_cli.subapps.applications.tools","title":"tools","text":"

    Provide tool functions for working with Application data.

    execute_application 
    execute_application(\n    app_module: JobbergateApplicationBase,\n    app_config: JobbergateApplicationConfig,\n    supplied_params: Optional[Dict[str, Any]] = None,\n    fast_mode: bool = False,\n)\n

    Execute the jobbergate application python module.

    Updates the app_config with values gathered in the question workflow

    :param: app_module: The source code for the application to execute :param: app_config: The configuration for the JobbergateApplication :param: supplied_params: Pre-set values for the parameters. Any questions about these values will be skipped. :param: fast_mode: If true, do not ask the user questions. Just use supplied_params or defaults :returns: The configuration values collected from the user by executing the application

    fetch_application_data 
    fetch_application_data(\n    jg_ctx: JobbergateContext,\n    id: Optional[int] = None,\n    identifier: Optional[str] = None,\n) -> ApplicationResponse\n

    Retrieve an application from the API by id or identifier.

    :param: jg_ctx: The JobbergateContext. Needed to access the Httpx client with which to make API calls :param: id: The id of the application to fetch :param: identifier: If supplied, look for an application instance with the provided identifier :returns: An instance of ApplicationResponse containing the application data

    get_upload_files 
    get_upload_files(application_path: pathlib.Path)\n

    Context manager to build the files parameter.

    Open the supplied file(s) and build a files param appropriate for using multi-part file uploads with the client.

    load_application_config_from_source 
    load_application_config_from_source(\n    config_source: str,\n) -> JobbergateApplicationConfig\n

    Load the JobbergateApplicationConfig from a text string containing the config as YAML.

    :param: config_source: The YAML containing the config :returns: A JobbergateApplicationConfig instance with the config values

    load_application_data 
    load_application_data(\n    app_data: ApplicationResponse,\n    application_source_file: str,\n) -> Tuple[\n    JobbergateApplicationConfig, JobbergateApplicationBase\n]\n

    Validates and loads the data for an application returned from the API's applications GET endpoint.

    As part of the Jobbergate data restructure, sections of the legacy jobbergate.yaml are now stored in different tables in the backend. This function reconstructs them from app_data.workflow_file.runtime_config and app_data.template_vars for backward compatibility.

    :param: app_data: A dictionary containing the application data :returns: A tuple containing the application config and the application module

    load_application_from_source 
    load_application_from_source(\n    app_source: str, app_config: JobbergateApplicationConfig\n) -> JobbergateApplicationBase\n

    Load the JobbergateApplication class from a text string containing the source file.

    Creates the module in a temporary file and imports it with importlib.

    Adapted from: https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly

    :param: app_source: The JobbergateApplication source code to load :param: app_config: The JobbergateApplicationConfig needed to instantiate the JobbergateApplication

    load_default_config 
    load_default_config() -> Dict[str, Any]\n

    Load the default config for an application.

    save_application_files 
    save_application_files(\n    jg_ctx: JobbergateContext,\n    application_data: ApplicationResponse,\n    destination_path: pathlib.Path,\n) -> List[pathlib.Path]\n

    Save the application files from the API response into a local destination.

    upload_application 
    upload_application(\n    jg_ctx: JobbergateContext,\n    application_path: pathlib.Path,\n    application_id: Optional[int],\n    application_identifier: Optional[str],\n) -> bool\n

    Upload an application given an application path and the application id.

    :param: jg_ctx: The JobbergateContext. Needed to access the Httpx client with which to make API calls :param: application_path: The directory where the application files to upload may be found :param: application_id: The id of the application for which to upload data :param: application_identifier: The identifier of the application for which to upload data :returns: True if the upload was successful; False otherwise

    "},{"location":"reference/cli/#jobbergate_cli.subapps.clusters","title":"clusters","text":"

    Provide a sub-app for interacting with Cluster data.

    "},{"location":"reference/cli/#jobbergate_cli.subapps.clusters.app","title":"app","text":"

    Provide a typer app that can interact with Cluster data in a cruddy manner.

    list_all 
    list_all(ctx: typer.Context)\n

    Show available clusters

    "},{"location":"reference/cli/#jobbergate_cli.subapps.clusters.tools","title":"tools","text":"

    Provide tool functions for working with Cluster data

    "},{"location":"reference/cli/#jobbergate_cli.subapps.job_scripts","title":"job_scripts","text":"

    Provide a sub-app for interacting with Job Script data.

    "},{"location":"reference/cli/#jobbergate_cli.subapps.job_scripts.app","title":"app","text":"

    Provide a typer app that can interact with Job Script data in a cruddy manner.

    delete 
    delete(\n    ctx: typer.Context,\n    id: int = typer.Option(\n        ...,\n        \"--id\",\n        \"-i\",\n        help=\"The id of the job script to delete\",\n    ),\n)\n

    Delete an existing job script.

    download_files 
    download_files(\n    ctx: typer.Context,\n    id: int = typer.Option(\n        ..., help=\"The specific id of the job script.\"\n    ),\n)\n

    Download the files from a job script to the current working directory.

    get_one 
    get_one(\n    ctx: typer.Context,\n    id: int = typer.Option(\n        ...,\n        \"--id\",\n        \"-i\",\n        help=\"The specific id of the job script.\",\n    ),\n)\n

    Get a single job script by id.

    list_all 
    list_all(\n    ctx: typer.Context,\n    show_all: bool = typer.Option(\n        False,\n        \"--all\",\n        help=\"Show all job scripts, even the ones owned by others\",\n    ),\n    search: Optional[str] = typer.Option(\n        None, help=\"Apply a search term to results\"\n    ),\n    sort_order: SortOrder = typer.Option(\n        SortOrder.UNSORTED, help=\"Specify sort order\"\n    ),\n    sort_field: Optional[str] = typer.Option(\n        None,\n        help=\"The field by which results should be sorted\",\n    ),\n    from_application_id: Optional[int] = typer.Option(\n        None,\n        help=\"Filter job-scripts by the application-id they were created from.\",\n    ),\n)\n

    Show available job scripts

    render 
    render(\n    ctx: typer.Context,\n    name: Optional[str] = typer.Option(\n        None,\n        \"--name\",\n        \"-n\",\n        help=dedent(\n            \"\\n            The name of the job script to create.\\n            If this is not supplied, the name will be derived from the base application.\\n            \"\n        ),\n    ),\n    application_id: Optional[int] = typer.Option(\n        None,\n        \"--application-id\",\n        \"-i\",\n        help=\"The id of the application from which to create the job script.\",\n    ),\n    application_identifier: Optional[str] = typer.Option(\n        None,\n        help=\"The identifier of the application from which to create the job script.\",\n    ),\n    description: Optional[str] = typer.Option(\n        None,\n        help=\"Optional text describing the job script.\",\n    ),\n    sbatch_params: Optional[List[str]] = typer.Option(\n        None,\n        help=\"Optional parameter to submit raw sbatch parameters.\",\n    ),\n    param_file: Optional[pathlib.Path] = typer.Option(\n        None,\n        help=dedent(\n            \"\\n            Supply a json file that contains the parameters for populating templates.\\n            If this is not supplied, the question asking in the application is triggered.\\n            \"\n        ),\n    ),\n    cluster_name: Optional[str] = typer.Option(\n        None,\n        help=\"The name of the cluster where the job should be submitted (i.g. 'nash-staging')\",\n    ),\n    execution_directory: Optional[\n        pathlib.Path\n    ] = typer.Option(\n        None,\n        help=dedent(\n            '\\n            The path on the cluster where the job script should be executed.\\n            If provided as a relative path, it will be converted as an absolute path from your current\\n            working directory. If you use \"~\" to denote your home directory, the path will be expanded to an\\n            absolute path for your home directory on *this* machine.\\n            '\n        ).strip(),\n    ),\n    download: Optional[bool] = typer.Option(\n        None,\n        help=\"Download the job script files to the current working directory\",\n    ),\n    fast: bool = typer.Option(\n        False,\n        \"--fast\",\n        \"-f\",\n        help=\"Use default answers (when available) instead of asking the user.\",\n    ),\n    submit: Optional[bool] = typer.Option(\n        None,\n        help=\"Do not ask the user if they want to submit a job.\",\n    ),\n)\n

    Render a new job script from an application.

    show_files 
    show_files(\n    ctx: typer.Context,\n    id: int = typer.Option(\n        ..., help=\"The specific id of the job script.\"\n    ),\n    plain: bool = typer.Option(\n        False, help=\"Show the files in plain text.\"\n    ),\n)\n

    Show the files for a single job script by id.

    update 
    update(\n    ctx: typer.Context,\n    id: int = typer.Option(\n        ...,\n        \"--id\",\n        \"-i\",\n        help=\"The id of the job script to update\",\n    ),\n    name: Optional[str] = typer.Option(\n        None, help=\"Optional new name of the job script.\"\n    ),\n    description: Optional[str] = typer.Option(\n        None,\n        help=\"Optional new text describing the job script.\",\n    ),\n)\n

    Update an existing job script.

    "},{"location":"reference/cli/#jobbergate_cli.subapps.job_scripts.tools","title":"tools","text":"

    Provide tool functions for working with Job Script data

    download_job_script_files 
    download_job_script_files(\n    id: int, jg_ctx: JobbergateContext\n) -> List[pathlib.Path]\n

    Download the job script files from the API and save them to the current working directory.

    fetch_job_script_data 
    fetch_job_script_data(\n    jg_ctx: JobbergateContext, id: int\n) -> JobScriptResponse\n

    Retrieve a job_script from the API by id

    flatten_param_dict 
    flatten_param_dict(\n    param_dict: Dict[str, Any]\n) -> Dict[str, Any]\n

    Flatten an input dictionary to support the rendering process.

    See the example:

    param_dict = { ... \"application_config\": {\"job_name\": \"rats\", \"partitions\": [...]}, ... \"jobbergate_config\": { ... \"default_template\": \"test_job_script.sh\", ... \"supporting_files\": [...], ... \"supporting_files_output_name\": {...}, ... \"template_files\": [...], ... \"job_script_name\": None, ... \"output_directory\": \".\", ... \"partition\": \"debug\", ... \"job_name\": \"rats\", ... }, ... } flat_param_dict = flatten_param_dict(param_dict) print(flat_param_dict) { \"job_name\": \"rats\", \"partitions\": [\"debug\", \"partition1\"], \"default_template\": \"test_job_script.sh\", \"supporting_files\": [\"test_job_script.sh\"], \"supporting_files_output_name\": {\"test_job_script.sh\": [...]}, \"template_files\": [\"templates/test_job_script.sh\"], \"job_script_name\": None, \"output_directory\": \".\", \"partition\": \"debug\", }

    get_template_output_name_mapping 
    get_template_output_name_mapping(\n    config: JobbergateConfig, job_name: str\n) -> Dict[str, str]\n

    Get the mapping of template names to output names.

    This provides the mapping as expected by the API v4 from the configuration on CLI v3.

    question_helper 
    question_helper(\n    question_func: Callable,\n    text: str,\n    default: Any,\n    fast: bool,\n    actual_value: Optional[Any],\n)\n

    Helper function for asking questions to the user.

    :param Callable question_func: The function to use to ask the question :param str text: The text of the question to ask :param Any default: The default value to use if the user does not provide one :param bool fast: Whether to use default answers (when available) instead of asking the user :param Any actual_value: The actual value provided by the user, if any

    :returns: actual_value or default or the value provided by the user

    The actual_value has the most priority and will be returned if it is not None. After evaluating the actual_value, the fast mode will determine if the default value will be used. Otherwise, the question will be prompted to the user.

    remove_prefix 
    remove_prefix(s: str) -> str\n

    Remove the prefix 'templates/' from a string

    remove_prefix_suffix 
    remove_prefix_suffix(s: str) -> str\n

    Remove the prefix 'templates/' and suffixes '.j2' and '.jinja2' from a string

    render_job_script 
    render_job_script(\n    jg_ctx: JobbergateContext,\n    name: Optional[str] = None,\n    application_id: Optional[int] = None,\n    application_identifier: Optional[str] = None,\n    description: Optional[str] = None,\n    sbatch_params: Optional[List[str]] = None,\n    param_file: Optional[pathlib.Path] = None,\n    fast: bool = False,\n) -> JobScriptResponse\n

    Render a new job script from an application.

    :param str name: Name of the new job script. :param Optional[int] application_id: Id of the base application. :param Optional[str] application_identifier: Identifier of the base application. :param Optional[str] description: Description of the new job script. :param Optional[List[str]] sbatch_params: List of sbatch parameters. :param Optional[pathlib.Path] param_file: Path to a parameters file. :param bool fast: Whether to use default answers (when available) instead of asking the user. :param JobbergateContext jg_ctx: The Jobbergate context. :return JobScriptResponse: The new job script.

    save_job_script_files 
    save_job_script_files(\n    jg_ctx: JobbergateContext,\n    job_script_data: JobScriptResponse,\n    destination_path: pathlib.Path,\n) -> List[pathlib.Path]\n

    Save the job script files from the API response to the output path.

    update_template_files_information 
    update_template_files_information(\n    app_data: ApplicationResponse,\n    app_config: JobbergateApplicationConfig,\n)\n

    Update the information about the template files if not already present in the configuration.

    upload_job_script_files 
    upload_job_script_files(\n    jg_ctx: JobbergateContext,\n    job_script_id: int,\n    job_script_path: pathlib.Path,\n    supporting_file_paths: Optional[\n        List[pathlib.Path]\n    ] = None,\n)\n

    Upload a job-script and its supporting files given their paths and the job-script id.

    :param: jg_ctx: The JobbergateContext. Needed to access the Httpx client with which to make API calls :param: job_script_path: The path to the job-script file to upload :param: supporting_file_paths: The paths to any supporting files to upload with the job-scritpt :param: job_script_id: The id of the job-script for which to upload data :returns: True if the main job script upload was successful; False otherwise

    validate_parameter_file 
    validate_parameter_file(\n    parameter_path: pathlib.Path,\n) -> Dict[str, Any]\n

    Validate parameter file at the supplied path and returns the parsed dict.

    Confirms

    parameter_path exists parameter_path is a valid json file

    "},{"location":"reference/cli/#jobbergate_cli.subapps.job_submissions","title":"job_submissions","text":"

    Provide a sub-app for interacting with Job Submission data.

    "},{"location":"reference/cli/#jobbergate_cli.subapps.job_submissions.app","title":"app","text":"

    Provide a typer app that can interact with Job Submission data in a cruddy manner.

    create 
    create(\n    ctx: typer.Context,\n    name: str = typer.Option(\n        ...,\n        \"--name\",\n        \"-n\",\n        help=\"The name of the job submission to create\",\n    ),\n    description: Optional[str] = typer.Option(\n        None,\n        help=\"A helpful description of the job submission\",\n    ),\n    job_script_id: int = typer.Option(\n        ...,\n        \"--job-script-id\",\n        \"-i\",\n        help=\"The id of the job_script from which to create the job submission\",\n    ),\n    cluster_name: str = typer.Option(\n        None,\n        help=\"The name of the cluster where the job should be submitted (i.g. 'nash-staging')\",\n    ),\n    execution_directory: Optional[Path] = typer.Option(\n        None,\n        help=dedent(\n            '\\n            The path on the cluster where the job script should be executed.\\n            If provided as a relative path, it will be converted as an absolute path from your current\\n            working directory. If you use \"~\" to denote your home directory, the path will be expanded to an\\n            absolute path for your home directory on *this* machine.\\n            '\n        ).strip(),\n    ),\n    execution_parameters: Optional[Path] = typer.Option(\n        None,\n        help=dedent(\n            \"\\n            The path to a JSON file containing the parameters to be passed to the job submission.\\n            See more details at: https://slurm.schedmd.com/rest_api.html\\n            \"\n        ).strip(),\n        exists=True,\n        readable=True,\n        resolve_path=True,\n    ),\n    download: bool = typer.Option(\n        False,\n        help=\"Download the job script files to the current working directory\",\n    ),\n)\n

    Create a new job submission.

    delete 
    delete(\n    ctx: typer.Context,\n    id: int = typer.Option(\n        ...,\n        \"--id\",\n        \"-i\",\n        help=\"The id of the job submission to delete\",\n    ),\n)\n

    Delete an existing job submission.

    get_one 
    get_one(\n    ctx: typer.Context,\n    id: int = typer.Option(\n        ...,\n        \"--id\",\n        \"-i\",\n        help=\"The specific id of the job submission.\",\n    ),\n)\n

    Get a single job submission by id

    list_all 
    list_all(\n    ctx: typer.Context,\n    show_all: bool = typer.Option(\n        False,\n        \"--all\",\n        help=\"Show all job submissions, even the ones owned by others\",\n    ),\n    search: Optional[str] = typer.Option(\n        None, help=\"Apply a search term to results\"\n    ),\n    sort_order: SortOrder = typer.Option(\n        SortOrder.UNSORTED, help=\"Specify sort order\"\n    ),\n    sort_field: Optional[str] = typer.Option(\n        None,\n        help=\"The field by which results should be sorted\",\n    ),\n    from_job_script_id: Optional[int] = typer.Option(\n        None,\n        help=\"Filter job-submissions by the job-script-id they were created from.\",\n    ),\n)\n

    Show available job submissions.

    "},{"location":"reference/cli/#jobbergate_cli.subapps.job_submissions.tools","title":"tools","text":"

    Provide tool functions for working with Job Submission data

    create_job_submission 
    create_job_submission(\n    jg_ctx: JobbergateContext,\n    job_script_id: int,\n    name: str,\n    description: Optional[str] = None,\n    cluster_name: Optional[str] = None,\n    execution_directory: Optional[Path] = None,\n    execution_parameters_file: Optional[Path] = None,\n) -> JobSubmissionResponse\n

    Create a Job Submission from the given Job Script.

    :param: jg_ctx: The JobbergateContext. Used to retrieve the client for requests and the email of the submitting user :param: job_script_id: The id of the Job Script to submit to Slurm :param: name: The name to attach to the Job Submission :param: description: An optional description that may be added to the Job Submission :param: cluster_name: An optional cluster_name for the cluster where the job should be executed, If left off, it will default to the DEFAULT_CLUSTER_NAME from the settings. If no default is set, an exception will be raised. :param: execution_directory: An optional directory where the job should be executed. If provided as a relative path, it will be constructed as an absolute path relative to the current working directory. :param: execution_parameters_file: An optional file containing the execution parameters for the job.

    :returns: The Job Submission data returned by the API after creating the new Job Submission

    fetch_job_submission_data 
    fetch_job_submission_data(\n    jg_ctx: JobbergateContext, job_submission_id: int\n) -> JobSubmissionResponse\n

    Retrieve a job submission from the API by id

    "},{"location":"reference/cli/#jobbergate_cli.text_tools","title":"text_tools","text":"

    Provide some basic tools for manipulating text.

    "},{"location":"reference/cli/#jobbergate_cli.text_tools.conjoin","title":"conjoin","text":"
    conjoin(*items: str, join_str: str = '\\n') -> str\n

    Joins strings supplied as args.

    Helper that wraps str.join() without having to pack strings in an iterable.

    "},{"location":"reference/cli/#jobbergate_cli.text_tools.copy_to_clipboard","title":"copy_to_clipboard","text":"
    copy_to_clipboard(text: str) -> bool\n

    Copy the provided text to the clipboard.

    If the clipboard is not available, return False. Otherwise, return True.

    "},{"location":"reference/cli/#jobbergate_cli.text_tools.dedent","title":"dedent","text":"
    dedent(text: str) -> str\n

    Dedents a paragraph after removing leading and trailing whitespace.

    "},{"location":"reference/cli/#jobbergate_cli.text_tools.dedent_all","title":"dedent_all","text":"
    dedent_all(*texts: str, join_str: str = '\\n') -> str\n

    Dedents each blob supplied as an argument and then joins them.

    "},{"location":"reference/cli/#jobbergate_cli.text_tools.indent","title":"indent","text":"
    indent(text: str, prefix: str = '    ', **kwargs) -> str\n

    Simple wrapper for the textwrap.indent() method but includes a default prefix.

    "},{"location":"reference/cli/#jobbergate_cli.text_tools.unwrap","title":"unwrap","text":"
    unwrap(text: str) -> str\n

    Unwraps a paragraph of text into a single line.

    The text may be indented.

    "},{"location":"reference/cli/#jobbergate_cli.time_loop","title":"time_loop","text":"

    Provide a time-loop class that can be used to to iterate during a given window of time.

    "},{"location":"reference/cli/#jobbergate_cli.time_loop.Tick","title":"Tick","text":"

    Bases: BaseModel

    A helper class describing a \"tick\".

    Contains a counter, elapsed time since the last tick, and total elapsed time.

    "},{"location":"reference/cli/#jobbergate_cli.time_loop.TimeLoop","title":"TimeLoop","text":"

    A special iterator that will iterate for a specified duration of time.

    Uses a progress meter to show the user how much time is left. Each iteration of the time-loop produces a tick.

    "},{"location":"reference/cli/#jobbergate_cli.time_loop.TimeLoop.__del__","title":"__del__","text":"
    __del__()\n

    Explicitly clear the progress meter if the time-loop is destroyed.

    "},{"location":"reference/cli/#jobbergate_cli.time_loop.TimeLoop.__init__","title":"__init__","text":"
    __init__(\n    duration: Union[pendulum.Duration, int],\n    message: str = \"Processing\",\n    color: str = \"green\",\n)\n

    Initialize the time-loop.

    Duration may be either a count of seconds or a pendulum.duration.

    "},{"location":"reference/cli/#jobbergate_cli.time_loop.TimeLoop.__iter__","title":"__iter__","text":"
    __iter__() -> TimeLoop\n

    Start the iterator.

    Creates and starts the progress meter

    "},{"location":"reference/cli/#jobbergate_cli.time_loop.TimeLoop.__next__","title":"__next__","text":"
    __next__() -> Tick\n

    Iterates the time loop and returns a tick.

    If the duration is complete, clear the progress meter and stop iteration.

    "},{"location":"reference/cli/#jobbergate_cli.time_loop.TimeLoop.clear","title":"clear","text":"
    clear()\n

    Clear the time-loop.

    Stops the progress meter (if it is set) and reset moments, counter, progress meter.

    "},{"location":"reference/core/","title":"Jobbergate Core Reference","text":""},{"location":"reference/core/#jobbergate_core","title":"jobbergate_core","text":"

    Jobbergate-core contains key components that are shared among sub-projects.

    "},{"location":"reference/core/#jobbergate_core.AuthenticationError","title":"AuthenticationError","text":"

    Bases: Buzz

    Base exception for errors related to authentication on Jobbergate.

    "},{"location":"reference/core/#jobbergate_core.JobbergateAuthHandler","title":"JobbergateAuthHandler dataclass","text":"

    High-level class used to manage authentication to requests to the Jobbergate-API

    After an instance of this class is created, it can be used to authenticate requests from both requests and httpx packages by passing it to the auth parameter on the request (see examples below).

    It just works out of the box. Behind the scenes, this procedure calls the :meth:JobbergateAuthHandler.acquire_access method to load the available tokens from the cache directory, it tries to refresh them if they are expired, or provides an URL to the user to login on the system.

    Notice all steps above are also available individually as public methods, allowing a fine control for advanced users.

    .. _requests: https://requests.readthedocs.io/en/latest/ .. _httpx: https://www.python-httpx.org/

    Parameters:

    Name Type Description Default cache_directory Path

    Directory to be used for the caching tokens.

    required login_domain str

    Domain used for the login.

    required login_audience str

    Audience of the login.

    required login_client_id str

    Client ID used for login.

    'default' Note

    These values depend on the identity provider used for authentication. Consult your system administrator or contact Omnivector support support@omnivector.solutions for further assistance.

    Note

    This class can interoperate with the tokens generated by the jobbergate-cli package, as long as they are stored in the same cache directory.

    Examples:

    The following example shows how to use the :meth:`JobbergateAuthHandler`\nclass to authenticate a request:\n\n>>> from pathlib import Path\n>>> import requests\n>>> from jobbergate_core import JobbergateAuthHandler\n>>> jobbergate_auth = JobbergateAuthHandler(\n...     cache_directory=Path(\".\"),\n...     login_domain=\"http://keycloak.local:8080/realms/jobbergate-local\",\n...     login_audience=\"https://local.omnivector.solutions\",\n...     login_client_id=\"cli\",\n... )\n>>> jobbergate_base_url = \"http://localhost:8000/jobbergate\"\n>>> response = requests.get(\n...     f\"{jobbergate_base_url}/applications\",\n...     auth=jobbergate_auth # this is the important part\n)\nLogin Here: http://keycloak.local:8080/realms/jobbergate-local/device?user_code=LMVJ-XOLG\n>>> response.raise_for_status()\n>>> print(f\"response = {response.json()}\")\n
    "},{"location":"reference/core/#jobbergate_core.JobbergateAuthHandler.__call__","title":"__call__","text":"
    __call__(request)\n

    This internal method allows the integration with the requests library.

    It is called automatically when the instance is passed to the auth parameter, see the examples in the class docstring.

    It adds the Authorization header to the request with the access token.

    "},{"location":"reference/core/#jobbergate_core.JobbergateAuthHandler.acquire_access","title":"acquire_access","text":"
    acquire_access() -> str\n

    High-level method to acquire a valid access token.

    This method will attempt, in order:

    • Use the internal access token from the instance
    • Load the tokens from the cache directory (see :meth:JobbergateAuthHandler.load_from_cache)
    • If the access token is unavailable or expired, refresh both tokens using the refresh token grant type (see :meth:JobbergateAuthHandler.refresh_tokens)
    • If the refresh token is unavailable or expired, login to generate both tokens (see :meth:JobbergateAuthHandler.login)

    Returns:

    Type Description str

    The bearer access token.

    Raises:

    Type Description AuthenticationError

    If all of the steps above fail to acquire a valid access token.

    "},{"location":"reference/core/#jobbergate_core.JobbergateAuthHandler.load_from_cache","title":"load_from_cache","text":"
    load_from_cache() -> None\n

    Load the tokens that are available at the cache directory.

    "},{"location":"reference/core/#jobbergate_core.JobbergateAuthHandler.login","title":"login","text":"
    login() -> None\n

    Login to Jobbergate.

    An URL will be printed to the console, the user must open it in a browser and provide their access credentials.

    After the login is completed, the tokens will be saved to the cache directory.

    "},{"location":"reference/core/#jobbergate_core.JobbergateAuthHandler.logout","title":"logout","text":"
    logout() -> None\n

    Logout from Jobbergate by clearing the loaded tokens and their cache on the disk.

    "},{"location":"reference/core/#jobbergate_core.JobbergateAuthHandler.refresh_tokens","title":"refresh_tokens","text":"
    refresh_tokens() -> None\n

    Refresh the tokens.

    After the refresh operation is completed, the tokens will be saved to the cache directory.

    Raises:

    Type Description AuthenticationError

    If the refresh token is missing or expired.

    "},{"location":"reference/core/#jobbergate_core.JobbergateAuthHandler.save_to_cache","title":"save_to_cache","text":"
    save_to_cache() -> None\n

    Save the tokens to the cache directory.

    Note

    This method will create the cache directory if it does not exist.

    "},{"location":"reference/core/#jobbergate_core.TokenError","title":"TokenError","text":"

    Bases: AuthenticationError

    Exception for errors related to tokens on Jobbergate.

    "},{"location":"reference/core/#jobbergate_core.auth","title":"auth","text":"

    Utilities for handling auth in Jobbergate.

    "},{"location":"reference/core/#jobbergate_core.auth.AuthenticationError","title":"AuthenticationError","text":"

    Bases: Buzz

    Base exception for errors related to authentication on Jobbergate.

    "},{"location":"reference/core/#jobbergate_core.auth.JobbergateAuthHandler","title":"JobbergateAuthHandler dataclass","text":"

    High-level class used to manage authentication to requests to the Jobbergate-API

    After an instance of this class is created, it can be used to authenticate requests from both requests and httpx packages by passing it to the auth parameter on the request (see examples below).

    It just works out of the box. Behind the scenes, this procedure calls the :meth:JobbergateAuthHandler.acquire_access method to load the available tokens from the cache directory, it tries to refresh them if they are expired, or provides an URL to the user to login on the system.

    Notice all steps above are also available individually as public methods, allowing a fine control for advanced users.

    .. _requests: https://requests.readthedocs.io/en/latest/ .. _httpx: https://www.python-httpx.org/

    Parameters:

    Name Type Description Default cache_directory Path

    Directory to be used for the caching tokens.

    required login_domain str

    Domain used for the login.

    required login_audience str

    Audience of the login.

    required login_client_id str

    Client ID used for login.

    'default' Note

    These values depend on the identity provider used for authentication. Consult your system administrator or contact Omnivector support support@omnivector.solutions for further assistance.

    Note

    This class can interoperate with the tokens generated by the jobbergate-cli package, as long as they are stored in the same cache directory.

    Examples:

    The following example shows how to use the :meth:`JobbergateAuthHandler`\nclass to authenticate a request:\n\n>>> from pathlib import Path\n>>> import requests\n>>> from jobbergate_core import JobbergateAuthHandler\n>>> jobbergate_auth = JobbergateAuthHandler(\n...     cache_directory=Path(\".\"),\n...     login_domain=\"http://keycloak.local:8080/realms/jobbergate-local\",\n...     login_audience=\"https://local.omnivector.solutions\",\n...     login_client_id=\"cli\",\n... )\n>>> jobbergate_base_url = \"http://localhost:8000/jobbergate\"\n>>> response = requests.get(\n...     f\"{jobbergate_base_url}/applications\",\n...     auth=jobbergate_auth # this is the important part\n)\nLogin Here: http://keycloak.local:8080/realms/jobbergate-local/device?user_code=LMVJ-XOLG\n>>> response.raise_for_status()\n>>> print(f\"response = {response.json()}\")\n
    "},{"location":"reference/core/#jobbergate_core.auth.JobbergateAuthHandler.__call__","title":"__call__","text":"
    __call__(request)\n

    This internal method allows the integration with the requests library.

    It is called automatically when the instance is passed to the auth parameter, see the examples in the class docstring.

    It adds the Authorization header to the request with the access token.

    "},{"location":"reference/core/#jobbergate_core.auth.JobbergateAuthHandler.acquire_access","title":"acquire_access","text":"
    acquire_access() -> str\n

    High-level method to acquire a valid access token.

    This method will attempt, in order:

    • Use the internal access token from the instance
    • Load the tokens from the cache directory (see :meth:JobbergateAuthHandler.load_from_cache)
    • If the access token is unavailable or expired, refresh both tokens using the refresh token grant type (see :meth:JobbergateAuthHandler.refresh_tokens)
    • If the refresh token is unavailable or expired, login to generate both tokens (see :meth:JobbergateAuthHandler.login)

    Returns:

    Type Description str

    The bearer access token.

    Raises:

    Type Description AuthenticationError

    If all of the steps above fail to acquire a valid access token.

    "},{"location":"reference/core/#jobbergate_core.auth.JobbergateAuthHandler.load_from_cache","title":"load_from_cache","text":"
    load_from_cache() -> None\n

    Load the tokens that are available at the cache directory.

    "},{"location":"reference/core/#jobbergate_core.auth.JobbergateAuthHandler.login","title":"login","text":"
    login() -> None\n

    Login to Jobbergate.

    An URL will be printed to the console, the user must open it in a browser and provide their access credentials.

    After the login is completed, the tokens will be saved to the cache directory.

    "},{"location":"reference/core/#jobbergate_core.auth.JobbergateAuthHandler.logout","title":"logout","text":"
    logout() -> None\n

    Logout from Jobbergate by clearing the loaded tokens and their cache on the disk.

    "},{"location":"reference/core/#jobbergate_core.auth.JobbergateAuthHandler.refresh_tokens","title":"refresh_tokens","text":"
    refresh_tokens() -> None\n

    Refresh the tokens.

    After the refresh operation is completed, the tokens will be saved to the cache directory.

    Raises:

    Type Description AuthenticationError

    If the refresh token is missing or expired.

    "},{"location":"reference/core/#jobbergate_core.auth.JobbergateAuthHandler.save_to_cache","title":"save_to_cache","text":"
    save_to_cache() -> None\n

    Save the tokens to the cache directory.

    Note

    This method will create the cache directory if it does not exist.

    "},{"location":"reference/core/#jobbergate_core.auth.Token","title":"Token dataclass","text":"

    Low-level class used to handling tokens.

    Parameters:

    Name Type Description Default cache_directory Path

    The directory used for cache.

    required label str

    The type of token.

    required content str

    The content of the token (default is \"\").

    ''

    Attributes:

    Name Type Description file_path Path

    The path to the file associated with the token. It is computed as <cache_directory>/<label>.token.

    data Dict[str, Any]

    Metadata decoded from the token's content are available in this dictionary. Expiration date and permissions are some examples of data that can be found.

    "},{"location":"reference/core/#jobbergate_core.auth.Token.bearer_token","title":"bearer_token property","text":"
    bearer_token: str\n

    Return the token with the Bearer prefix.

    "},{"location":"reference/core/#jobbergate_core.auth.Token.__post_init__","title":"__post_init__","text":"
    __post_init__()\n

    Post init method.

    "},{"location":"reference/core/#jobbergate_core.auth.Token.clear_cache","title":"clear_cache","text":"
    clear_cache() -> None\n

    Clear the token from cache by removing the file associated with it.

    "},{"location":"reference/core/#jobbergate_core.auth.Token.is_expired","title":"is_expired","text":"
    is_expired() -> bool\n

    Check if the token is expired.

    Returns:

    Type Description bool

    True if the token is expired, False otherwise.

    Raises:

    Type Description TokenError

    If the expiration date is not found.

    "},{"location":"reference/core/#jobbergate_core.auth.Token.is_valid","title":"is_valid","text":"
    is_valid() -> bool\n

    Verify if the token is valid, i.e., has content and is not expired.

    "},{"location":"reference/core/#jobbergate_core.auth.Token.load_from_cache","title":"load_from_cache","text":"
    load_from_cache() -> Token\n

    Load the token from the cache directory.

    Parameters:

    Name Type Description Default cache_directory

    The path to the cache directory.

    required label

    The type of token.

    required

    Returns:

    Type Description Token

    A new token with the content replaced.

    "},{"location":"reference/core/#jobbergate_core.auth.Token.replace","title":"replace","text":"
    replace(**changes) -> Token\n

    Create a new instance of the token with the changes applied.

    Other Parameters:

    Name Type Description content

    The content of the token.

    cache_directory

    The directory containing the cache.

    label

    The type of token.

    "},{"location":"reference/core/#jobbergate_core.auth.Token.save_to_cache","title":"save_to_cache","text":"
    save_to_cache() -> None\n

    Save the token to the cache file associated with it.

    Raises:

    Type Description TokenError

    If the parent directory does not exist.

    TokenError

    If there is an unknown error while saving the token.

    "},{"location":"reference/core/#jobbergate_core.auth.TokenError","title":"TokenError","text":"

    Bases: AuthenticationError

    Exception for errors related to tokens on Jobbergate.

    "},{"location":"reference/core/#jobbergate_core.auth.TokenType","title":"TokenType","text":"

    Bases: str, Enum

    The types of tokens available in the system are access and refresh.

    "},{"location":"reference/core/#jobbergate_core.auth.exceptions","title":"exceptions","text":""},{"location":"reference/core/#jobbergate_core.auth.exceptions.AuthenticationError","title":"AuthenticationError","text":"

    Bases: Buzz

    Base exception for errors related to authentication on Jobbergate.

    "},{"location":"reference/core/#jobbergate_core.auth.exceptions.TokenError","title":"TokenError","text":"

    Bases: AuthenticationError

    Exception for errors related to tokens on Jobbergate.

    "},{"location":"reference/core/#jobbergate_core.auth.handler","title":"handler","text":"

    Utilities for handling authentication in the Jobbergate system.

    "},{"location":"reference/core/#jobbergate_core.auth.handler.JobbergateAuthHandler","title":"JobbergateAuthHandler dataclass","text":"

    High-level class used to manage authentication to requests to the Jobbergate-API

    After an instance of this class is created, it can be used to authenticate requests from both requests and httpx packages by passing it to the auth parameter on the request (see examples below).

    It just works out of the box. Behind the scenes, this procedure calls the :meth:JobbergateAuthHandler.acquire_access method to load the available tokens from the cache directory, it tries to refresh them if they are expired, or provides an URL to the user to login on the system.

    Notice all steps above are also available individually as public methods, allowing a fine control for advanced users.

    .. _requests: https://requests.readthedocs.io/en/latest/ .. _httpx: https://www.python-httpx.org/

    Parameters:

    Name Type Description Default cache_directory Path

    Directory to be used for the caching tokens.

    required login_domain str

    Domain used for the login.

    required login_audience str

    Audience of the login.

    required login_client_id str

    Client ID used for login.

    'default' Note

    These values depend on the identity provider used for authentication. Consult your system administrator or contact Omnivector support support@omnivector.solutions for further assistance.

    Note

    This class can interoperate with the tokens generated by the jobbergate-cli package, as long as they are stored in the same cache directory.

    Examples:

    The following example shows how to use the :meth:`JobbergateAuthHandler`\nclass to authenticate a request:\n\n>>> from pathlib import Path\n>>> import requests\n>>> from jobbergate_core import JobbergateAuthHandler\n>>> jobbergate_auth = JobbergateAuthHandler(\n...     cache_directory=Path(\".\"),\n...     login_domain=\"http://keycloak.local:8080/realms/jobbergate-local\",\n...     login_audience=\"https://local.omnivector.solutions\",\n...     login_client_id=\"cli\",\n... )\n>>> jobbergate_base_url = \"http://localhost:8000/jobbergate\"\n>>> response = requests.get(\n...     f\"{jobbergate_base_url}/applications\",\n...     auth=jobbergate_auth # this is the important part\n)\nLogin Here: http://keycloak.local:8080/realms/jobbergate-local/device?user_code=LMVJ-XOLG\n>>> response.raise_for_status()\n>>> print(f\"response = {response.json()}\")\n
    __call__ 
    __call__(request)\n

    This internal method allows the integration with the requests library.

    It is called automatically when the instance is passed to the auth parameter, see the examples in the class docstring.

    It adds the Authorization header to the request with the access token.

    acquire_access 
    acquire_access() -> str\n

    High-level method to acquire a valid access token.

    This method will attempt, in order:

    • Use the internal access token from the instance
    • Load the tokens from the cache directory (see :meth:JobbergateAuthHandler.load_from_cache)
    • If the access token is unavailable or expired, refresh both tokens using the refresh token grant type (see :meth:JobbergateAuthHandler.refresh_tokens)
    • If the refresh token is unavailable or expired, login to generate both tokens (see :meth:JobbergateAuthHandler.login)

    Returns:

    Type Description str

    The bearer access token.

    Raises:

    Type Description AuthenticationError

    If all of the steps above fail to acquire a valid access token.

    load_from_cache 
    load_from_cache() -> None\n

    Load the tokens that are available at the cache directory.

    login 
    login() -> None\n

    Login to Jobbergate.

    An URL will be printed to the console, the user must open it in a browser and provide their access credentials.

    After the login is completed, the tokens will be saved to the cache directory.

    logout 
    logout() -> None\n

    Logout from Jobbergate by clearing the loaded tokens and their cache on the disk.

    refresh_tokens 
    refresh_tokens() -> None\n

    Refresh the tokens.

    After the refresh operation is completed, the tokens will be saved to the cache directory.

    Raises:

    Type Description AuthenticationError

    If the refresh token is missing or expired.

    save_to_cache 
    save_to_cache() -> None\n

    Save the tokens to the cache directory.

    Note

    This method will create the cache directory if it does not exist.

    "},{"location":"reference/core/#jobbergate_core.auth.token","title":"token","text":"

    Utilities for handling tokens on Jobbergate.

    "},{"location":"reference/core/#jobbergate_core.auth.token.Token","title":"Token dataclass","text":"

    Low-level class used to handling tokens.

    Parameters:

    Name Type Description Default cache_directory Path

    The directory used for cache.

    required label str

    The type of token.

    required content str

    The content of the token (default is \"\").

    ''

    Attributes:

    Name Type Description file_path Path

    The path to the file associated with the token. It is computed as <cache_directory>/<label>.token.

    data Dict[str, Any]

    Metadata decoded from the token's content are available in this dictionary. Expiration date and permissions are some examples of data that can be found.

    bearer_token property 
    bearer_token: str\n

    Return the token with the Bearer prefix.

    __post_init__ 
    __post_init__()\n

    Post init method.

    clear_cache 
    clear_cache() -> None\n

    Clear the token from cache by removing the file associated with it.

    is_expired 
    is_expired() -> bool\n

    Check if the token is expired.

    Returns:

    Type Description bool

    True if the token is expired, False otherwise.

    Raises:

    Type Description TokenError

    If the expiration date is not found.

    is_valid 
    is_valid() -> bool\n

    Verify if the token is valid, i.e., has content and is not expired.

    load_from_cache 
    load_from_cache() -> Token\n

    Load the token from the cache directory.

    Parameters:

    Name Type Description Default cache_directory

    The path to the cache directory.

    required label

    The type of token.

    required

    Returns:

    Type Description Token

    A new token with the content replaced.

    replace 
    replace(**changes) -> Token\n

    Create a new instance of the token with the changes applied.

    Other Parameters:

    Name Type Description content

    The content of the token.

    cache_directory

    The directory containing the cache.

    label

    The type of token.

    save_to_cache 
    save_to_cache() -> None\n

    Save the token to the cache file associated with it.

    Raises:

    Type Description TokenError

    If the parent directory does not exist.

    TokenError

    If there is an unknown error while saving the token.

    "},{"location":"reference/core/#jobbergate_core.auth.token.TokenType","title":"TokenType","text":"

    Bases: str, Enum

    The types of tokens available in the system are access and refresh.

    "},{"location":"reference/core/#jobbergate_core.version","title":"version","text":"

    Provide the version of the package.

    "}]} \ No newline at end of file diff --git a/sitemap.xml.gz b/sitemap.xml.gz index f16a4b7f5c9b74795aaafdc63a8fe9c90ef1c253..13b59794cfe1119be355e1c1ba8a6410081b48c5 100644 GIT binary patch delta 13 Ucmb=gXP58h;CS*Ybs~EO03lfg6951J delta 13 Ucmb=gXP58h;JEiAdLnxT03ibf0RR91