-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathindex.xml
1051 lines (650 loc) · 65.6 KB
/
index.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>I Love To Delete Code</title>
<link>http://www.ilovetodeletecode.com/index.xml</link>
<description>Recent content on I Love To Delete Code</description>
<generator>Hugo -- gohugo.io</generator>
<language>en-us</language>
<lastBuildDate>Mon, 17 Feb 2020 22:35:15 +0100</lastBuildDate>
<atom:link href="http://www.ilovetodeletecode.com/index.xml" rel="self" type="application/rss+xml" />
<item>
<title>Resetting your HP BIOS password by modifying the main BIOS chip</title>
<link>http://www.ilovetodeletecode.com/post/2020/02/17/resetting-your-hp-bios-password-by-modifying-the-main-bios-chip/</link>
<pubDate>Mon, 17 Feb 2020 22:35:15 +0100</pubDate>
<guid>http://www.ilovetodeletecode.com/post/2020/02/17/resetting-your-hp-bios-password-by-modifying-the-main-bios-chip/</guid>
<description><p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;My inner security expert turned on the BIOS password on an HP laptop (EliteBook 840 G1), right after I bought it, around 6 years ago. Of course I forgot what the password was after a while, since I did not have a reason to change any of the settings, but this changed recently. I really wanted to play with <a href="https://docs.microsoft.com/en-us/windows/wsl/wsl2-index">WSL2</a> (Windows subtype for Linux) and Docker for Windows, and one of the prerequisites for both is to <a href="https://docs.microsoft.com/en-us/windows/wsl/wsl2-faq">enable Hyper-V</a> which is a setting in BIOS.<br />
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Is there a way I can remove the BIOS password? This blog post describes my attempts which have finally resulted in success.</p>
<p></p>
<h2 id="solutions-i-decided-not-to-go-for">Solutions I decided not to go for</h2>
<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;These are the solutions I have considered, but ended up not going for, because they weren&rsquo;t reasonable or were simply no longer applicable (due to the discountinued HP support). Note, that both item 4 and the solution I ended up with need the BIOS chip to be desoldered. This might sound scary, but it&rsquo;s actually quite simple with a little bit of hardware skills and the necessary equipment (or having a friend with both).</p>
<ol>
<li>Contact official HP support</li>
<li>Buy a new laptop</li>
<li>HP BIOS Unlock</li>
<li>Buy an unlocked BIOS chip on Ebay</li>
</ol>
<h5 id="contact-official-hp-support">Contact official HP support</h5>
<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;This was the way to go until <a href="https://support.hp.com/us-en/document/c06368824">HP decided to stop providing the solution</a> sometime in the middle of 2019. The process was to send HP the serial number of your laptop, for which they would generate a SMC.bin file and provide the instructions how to override the existing BIOS. Note that any SMC.bin files you will find on the internet, will not work for you, because the serial numbers will not match. I assume the serial number in the binary file is encrypted, which makes it impossible for you to replace.</p>
<h5 id="buy-a-new-laptop">Buy a new laptop</h5>
<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;To much of an overkill, even though the <a href="https://support.hp.com/gb-en/document/c03961746">laptop</a> is 6 years old I7 with 16 GB of RAM it is still in mint condition and does everything I need.</p>
<h5 id="hp-bios-unlock-tool">HP BIOS Unlock tool</h5>
<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;There is a <a href="http://mazzifsoftware.blogspot.com/p/hp-bios-unlock.html">tool</a> that works with some HP models, but not all of them. I think the newer HP models have extra security that does not allow you to flash the BIOS just like this tool does.<br />
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;This is not the same tool as mentioned and used later in the solution.</p>
<h5 id="buy-unlocked-bios-chip-on-ebay">Buy unlocked BIOS chip on Ebay</h5>
<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;This involves replacing the main BIOS chip on your motherboard with an unlocked one you buy on Ebay. In fact the seller recommended replacing two chips. Main BIOS chip + EC (Embed Controller) chip, but they don&rsquo;t explain why.<br />
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;You also need to let them know the SKU, serial number and the model of your laptop. BIOS image dumps are not transferable, you can&rsquo;t use the same dump on two machines - even if they are of exactly the same model. They will take this information to <a href="https://www.win-raid.com/t1658f39-Guide-Clean-Dumped-Intel-Engine-CS-ME-CS-TXE-Regions-with-Data-Initialization.html">initialize the ME region (Data section)</a>.<br />
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;I have not decided to go for this solution because I found a more elegant one. Solution presented below will specifically target only the BIOS password and will leave the ME Data region unchanged.</p>
<h2 id="solution">Solution</h2>
<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;In the end I decided to go with the least invasive solution that doesn&rsquo;t involve changing any hardware parts or making massive modifications to the BIOS dump file.<br />
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;None of the work in this post is my own. This post is just an aggregate of everything I learned during my research, that gave me the understanding and confidence I needed to get the job done. The amount of information you can find about the hardware components is incredible. Next time one of my gadgets breaks, I will definitely put more effort into trying to get it fixed rather than throwing it away and buying a new one :). You can find all the links to hardware forums used in this project down below.</p>
<p>The final solution includes:</p>
<ol>
<li>Identifying the correct chip</li>
<li>Desoldering main bios chip</li>
<li>Modifying the content</li>
<li>Putting the main BIOS chip back</li>
</ol>
<h5 id="identifying-the-correct-chip">Identifying the correct chip</h5>
<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;There are two BIOS flash memory chips and first we need to identify which out of the two we need. I&rsquo;m not sure why there are motherboards with multiple BIOS chips. Separation of concerns, I guess. Anyone interested in this can read about it <a href="http://www.bios-chip24.com/epages/63730052.sf/en_GB/?ObjectPath=/Shops/63730052/Categories/Informationen_zum_Bios/Bios_Chip_lokalisierung">here</a>. As I already mentioned, some sellers on Ebay are offering 2 unlocked bios chips, but after reading up online, I learnt this is not necessary.<br />
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;In my case the correct chip was marked with a red dot, and was located right above the CMOS battery.</p>
<figure >
<a href="http://www.ilovetodeletecode.com/img/chip_location.png">
<img src="http://www.ilovetodeletecode.com/img/chip_location_small.png" alt="main BIOS memory chip above the CMOS battery" />
</a>
<figcaption>
<p>
main BIOS memory chip above the CMOS battery
</p>
</figcaption>
</figure>
<figure >
<a href="http://www.ilovetodeletecode.com/img/winbond_25q128fvsq.jpg">
<img src="http://www.ilovetodeletecode.com/img/winbond_25q128fvsq_small.jpg" alt="Winbond 25Q128FVSQ-1318" />
</a>
<figcaption>
<p>
Winbond 25Q128FVSQ-1318
</p>
</figcaption>
</figure>
<h5 id="desoldering-main-bios-chip">Desoldering main BIOS chip</h5>
<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;This step is easy if you have the right tools and experience (it took a hobby enthusiast 10 minutes). Place the chip into the adapter and plug it into an EEPROM programmer. In my case the main BIOS chip was a &ldquo;Winbond 25Q128FVSQ-1318&rdquo; and when copied its contents I got a dump of size 16MB. You can find all of the equipment used in the post listed below. Make sure you keep a copy of your original BIOS dump&hellip;just in case!!</p>
<figure >
<a href="http://www.ilovetodeletecode.com/img/Programmer_Adapter_Socket.png">
<img src="http://www.ilovetodeletecode.com/img/Programmer_Adapter_Socket_small.png" alt="SOIC8 SOP8 to DIP8 adapter for the BIOS chip" />
</a>
<figcaption>
<p>
SOIC8 SOP8 to DIP8 adapter for the BIOS chip
</p>
</figcaption>
</figure>
<h5 id="modifying-the-content">Modifying the content</h5>
<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;It is easy if you know what you need to do. In my case I had to clear 87 bytes starting at address 0DB3470. I learned this on <a href="https://forums.mydigitallife.net/threads/solved-hp-elitebook-840-g1-bios-admin-password.61719/">HP EliteBook 840 G1 BIOS Admin Password</a>, there is also an elegant Python application called <a href="https://www.badcaps.net/forum/showthread.php?t=79956">HP BIOS unlock tool</a>. I ended up using the Python script. The application takes a single argument which should be the path to your 16MB BIOS image. After running it, the script should create an unlocked version of your original image. You can view the result in the hex editor below (unlocked vs. locked BIOS dump content).</p>
<figure >
<a href="http://www.ilovetodeletecode.com/img/bios_dump.png">
<img src="http://www.ilovetodeletecode.com/img/bios_dump.png" />
</a>
</figure>
<h5 id="putting-the-main-bios-chip-back">Putting the main BIOS chip back</h5>
<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Not much to add here. Clean the pads a bit and solder the chip back. Make sure you pay attention to markings on the motherboard in order to connect pin 1 correctly (or just take a picture of the chip&rsquo;s rotation before you desolder it).</p>
<p>That&rsquo;s it, you have successfully unlocked your BIOS.</p>
<h2 id="bitlocker">Bitlocker</h2>
<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;First thing I did after removing the BIOS password was upgrading the BIOS version (you can&rsquo;t upgrade your BIOS if your BIOS is locked). I was swapping between two primary hard-drives and after I re-connected the bootable drive that was encrypted using Bitlocker I could not boot anymore without providing the Bitlocker backup key.<br />
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;It was either upgrading the BIOS or modifying the BIOS dump that changed the fingerprint of my laptop, so make sure you have the Bitlocker backup key ready in case your hard drive is encrypted using this technology.</p>
<h2 id="equipment-used">Equipment Used</h2>
<ul>
<li><strong>Adapter</strong> - SOIC8 SOP8 to DIP8 Wide-body Seat Wide 150mil 200mil Programmer Adapter</li>
<li><strong>Programmer</strong> - TL866A Tall Speed in-circuit Programmer USB</li>
<li><strong>Soldering station</strong> - soldering iron + hot air gun</li>
<li><a href="https://www.badcaps.net/forum/showthread.php?t=79956">HP Unlocking tool</a></li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;It took me a while to realise the main BIOS chip will have to be taken off the motherboard. My main concern was this &ldquo;surgical&rdquo; procedure would render the laptop useless. But it turned out that the 8 pin memory chip is quite big, easy to work with and located separately from all critical components. And if there had been a problem with the modified BIOS dump, I could just copy back the original content I backed up before the modification.</p>
<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Two online resources I got a lot of information from:</p>
<ul>
<li><a href="https://www.badcaps.net/forum/">Badcaps Forums - Salvation For Your Electronics!</a></li>
<li><a href="https://forums.mydigitallife.net/">My Digital Life Forums</a></li>
</ul></description>
</item>
<item>
<title>About</title>
<link>http://www.ilovetodeletecode.com/about/</link>
<pubDate>Thu, 30 Jan 2020 20:10:55 +0000</pubDate>
<guid>http://www.ilovetodeletecode.com/about/</guid>
<description>
<h2 id="about-me">About me</h2>
<p>My name is Jernej Gorički. I graduated from college in Faculty of Computer and Information Science, Ljubljana Slovenia. With background knowledge of C/C++ my weapon of choice are C# &amp; Java . I worked as a freelancer with focus on desktop and web applications for exclusive customers in Automation and Test &amp; Measurement Industry like <a href="http://www.metrel.si/">Metrel d.d.</a>, <a href="http://www.iskraimpuls.si/en/">Iskra Impuls d.o.o.</a> and <a href="http://www.korona.si/en/">Korona engineering d.d.</a>.</p>
<p>In 2015 I moved to London and started working in Legal and Finance Industry. At the moment I work for <a href="https://www.firstderivatives.com/">First Derivatives</a> as a senior developer.</p>
<h2 id="about-this-blog">About this blog</h2>
<p>For many artists the act of destroying their work played an important role in their career. Embarrassing early creations they would like to forget about, artistic rebirth and doubt in their work all left them remains (ideas) which they would gradually incorporate into new pieces.</p>
<p>What do artists have in common with developers? Metaphors play a key role in constructing concepts and terminology especially in the world of Object Oriented Programming. They also give you insight into programming problems and processes. As the title of a <a href="https://en.wikipedia.org/wiki/The_Art_of_Computer_Programming">fundamental monograph</a> implies computer programming is an art hence developers are all artists.</p>
<blockquote>
<p>The only thing I love to do more than write code is delete code. Deleting code brings me inner peace and joy. Deleting code is violating the law of entropy and setting the universe right.</p>
</blockquote>
</description>
</item>
<item>
<title>Running Apache Solr 6.5 on an Azure App Service instance</title>
<link>http://www.ilovetodeletecode.com/post/2017/06/01/running-apache-solr-6.5-on-an-azure-app-service-instance/</link>
<pubDate>Thu, 01 Jun 2017 22:35:15 +0100</pubDate>
<guid>http://www.ilovetodeletecode.com/post/2017/06/01/running-apache-solr-6.5-on-an-azure-app-service-instance/</guid>
<description>
<p>This post describes a daring way to attempt to deploy a single Solr node to an Azure App Service. I say daring because I could not find a copy paste solution on Stack Overflow. I saw some light at the end of the tunnel after reading the <a href="https://docs.microsoft.com/en-us/azure/app-service-web/web-sites-java-custom-upload">Upload a custom Java web app to Azure</a> article on Azure documentation which mentions Jetty and since Solr is running in a Jetty Servlet container by default I guessed I should at least give it a try.</p>
<h2 id="why-azure-app-services">Why Azure App Services?</h2>
<p>I have a small amount of data to index and I will never need more than one Solr node running, so why not. A pre-configured Linux VM would be an overkill. I haven&rsquo;t tried Azure Search yet, but with Solr I can retrieve all the stats data for my filters and search results in a single request, not to mention its powerful natural language processing capabilities.</p>
<p>Let&rsquo;s get to work&hellip;</p>
<h2 id="azure-portal-s-configuration-ui">Azure Portal&rsquo;s configuration UI</h2>
<p>You only need to enable Java (at least version 1.8 for Solr &gt; 6.x) and Jetty as a Web Container.</p>
<figure >
<a href="http://www.ilovetodeletecode.com/img/Post5_azure_portal_configuration.png">
<img src="http://www.ilovetodeletecode.com/img/Post5_azure_portal_configuration.png" />
</a>
</figure>
<h2 id="web-config-httpplatform-configuration">Web.config httpPlatform configuration</h2>
<p>Extract contents of <a href="http://www-eu.apache.org/dist/lucene/solr/6.5.1/">solr-6.5.1.zip</a> to <b>%HOME%\site\wwwroot</b> folder of your App Service and add a Web.config with the following settings:</p>
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<system.webServer>
<handlers>
<add name="httppPlatformHandler"
path="*"
verb="*"
modules="httpPlatformHandler"
resourceType="Unspecified" />
</handlers>
<httpPlatform processPath="%HOME%\site\wwwroot\bin\solr.cmd"
arguments="start -p %HTTP_PLATFORM_PORT%"
startupTimeLimit="20"
startupRetryCount="10"
stdoutLogEnabled="true">
</httpPlatform>
</system.webServer>
</configuration>
<p><code>HTTP_PLATFORM_PORT</code> is the environment variable that holds the port number Azure has dynamically assigned to the Java process. We use its value as a parameter to the <b>-p</b> switch so that Solr does not start on the default port which is <b>8983</b>.</p>
<p>Having <code>stdoutLogEnabled</code> enabled causes the <b>stdout</b> and <b>stderr</b> for the process specified in the <b>processPath</b> setting to be redirected to location defined in <b>stdoutLogFile</b>.</p>
<p>The default value of <b>stdoutLogFile</b> should be <b> %HOME%\LogFiles\ </b> but in my case the logs were actually written to <b>%HOME%\site\wwwroot\ </b></p>
<p>Although there seem to be some problems during the startup, Solr starts up successfully and you can access the Admin panel at &lt;yoursite&gt;.azurewebsites.net</p>
Access is denied.
Waiting up to 30 to see Solr running on port 16055
INFO - 2017-06-09 12:44:49.394; org.apache.http.impl.client.DefaultRequestDirector; I/O exception (java.net.SocketException) caught when connecting to {}->http://localhost:16055: Permission denied: connect
INFO - 2017-06-09 12:44:49.521; org.apache.http.impl.client.DefaultRequestDirector; Retrying connect to {}->http://localhost:16055
INFO - 2017-06-09 12:44:49.659; org.apache.http.impl.client.DefaultRequestDirector; I/O exception (java.net.SocketException) caught when connecting to {}->http://localhost:16055: Permission denied: connect
INFO - 2017-06-09 12:44:49.659; org.apache.http.impl.client.DefaultRequestDirector; Retrying connect to {}->http://localhost:16055
Started Solr server on port 16055. Happy searching!
<p>That&rsquo;s it! And oh yeah, don&rsquo;t forget to enable Authentication (Azure App Service / Solr Authentication Plugins) or else everyone will have access to your Admin panel and Solr API.</p>
</description>
</item>
<item>
<title>FileOptions Enumeration, Windows Cache Manager and Memory-Mapped Files (C#) Part 1</title>
<link>http://www.ilovetodeletecode.com/post/2014/11/16/fileoptions-enumeration-windows-cache-manager-and-memory-mapped-files-csharp-part-1/</link>
<pubDate>Sun, 16 Nov 2014 14:03:50 +0000</pubDate>
<guid>http://www.ilovetodeletecode.com/post/2014/11/16/fileoptions-enumeration-windows-cache-manager-and-memory-mapped-files-csharp-part-1/</guid>
<description><p>Because working with files is such a common and potentially expensive operation, every operating system introduces numerous levels of indirection when serving I/O requests. This is necessary in order to execute I/O operations in a reasonable time frame. But managing complexity through abstraction of various hardware and software layers that interact during file operations also has its disadvantages: you get a false impression that you don&rsquo;t need to know what is really going on under the hood :)</p>
<p>I needed to optimize a part of my code that was writing data to random locations in a binary file. After realizing that <a href="http://msdn.microsoft.com/en-us/library/system.io.fileoptions(v=vs.110).aspx">FileOptions.RandomAccess</a> does not automagically do the trick and make seeks (in a loop!) run noticeably faster, I started to dig into the FileOptions Enumeration. A couple of days later, bursting with knowledge on internal working of Windows Cache Manager and speeding up random writes using <a href="http://msdn.microsoft.com/en-us/library/dd997372%28v=vs.110%29.aspx">Memory-Mapped Files</a> I decided to write a two part blog post about it.</p>
<p></p>
<h2 id="fileoptions-enumeration-the-5-cache-flags">FileOptions Enumeration &amp; The 5 Cache Flags</h2>
<p>The description for FileOptions.RandomAccess on MSDN is pretty vague:</p>
<blockquote>
<p>&ldquo;FileOptions.RandomAccess indicates that the file is accessed randomly. The system can use this as a hint to optimize file caching.&rdquo;</p>
</blockquote>
<p>Even a general search for FileOptions.RandomAccess keyword did not result in any resources that would describe the attribute in a more detailed manner. With a little help of <a href="http://referencesource.microsoft.com/#mscorlib/system/io/filestream.cs,76ef6c04de9d0ed8">.NET Reference Source</a> you can quickly check that the FileOptions parameter you apply to FileStream&rsquo;s constructor maps to next to last parameter (dwFlagsAndAttributes) of the native <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/aa363858(v=vs.85).aspx">CreateFile function</a>. Lets check the flags from the native point of view.</p>
<table>
<thead>
<tr>
<th>.NET</th>
<th>Native</th>
</tr>
</thead>
<tbody>
<tr>
<td>FileOptions.RandomAccess</td>
<td>FILE_FLAG_RANDOM_ACCESS</td>
</tr>
<tr>
<td>FileOptions.SequentialScan</td>
<td>FILE_FLAG_SEQUENTIAL_SCAN</td>
</tr>
<tr>
<td>FileOptions.WriteThrough</td>
<td>FILE_FLAG_WRITE_THROUGH</td>
</tr>
<tr>
<td>FileOptions.NoBuffering</td>
<td>FILE_FLAG_NO_BUFFERING</td>
</tr>
<tr>
<td>FileAttributes.Temporary</td>
<td>FILE_ATTRIBUTE_TEMPORARY</td>
</tr>
</tbody>
</table>
<figure >
<img src="http://www.ilovetodeletecode.com/img/Post4_cache_layers.png" />
</figure>
<p>These flags affect caching - not the FileStream&rsquo;s internal buffer mechanism, but caching on the OS level (Cache Manager). The picture on your right (borrowed from this <a href="http://research.microsoft.com/apps/pubs/default.aspx?id=64538">research paper</a>) clearly shows that caching is a two stage buffering process. Every time you are reading/writing in .NET you are actually copying bytes between the two buffers. With this in mind it is also easier to understand the <a href="http://stackoverflow.com/questions/4921498/whats-the-difference-between-filestream-flush-and-filestream-flushtrue">difference between Flush() and Flush(true)</a>.</p>
<h2 id="windows-cache-manager-file-flag-no-buffering">Windows Cache Manager &amp; FILE_FLAG_NO_BUFFERING</h2>
<p>Each time you write or read data to/from disk drives the OS caches data unless FILE_FLAG_NO_BUFFERING is specified. In latter case your application has total control over the buffering process which is known as unbuffered file I/O. When inspecting .NET Reference Source 4.5.2 you can see that the FileOptions.NoBuffering enum is currently commented out. But the guard clause that validates FileOptions flags before calling native CreateFile function allows the value 0x20000000 and this value relates to the native flag FILE_FLAG_NO_BUFFERING. So the following initialization of FileStream would be legal:</p>
var f = new FileStream("c:\\file.txt", FileMode.Open, FileAccess.Read, FileShare.None, bufferSize, (FileOptions)0x20000000);
<p>Files opened with FILE_FLAG_NO_BUFFERING do not go through the Cache Manager (all of the cache flags are ignored) and working with such files requires from our application to follow some rules regarding buffer alignment criteria. This is an <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/cc644950(v=vs.85).aspx">advanced topic</a> and goes beyond this post. For each file that is accessed through the Cache Manager an array of internal data structures need to be created and managed. The bigger the file, the more resources it takes to successfully support caching. That is why it&rsquo;s advised to use FILE_FLAG_NO_BUFFERING when working with very large files.</p>
<h2 id="cache-manager-routines">Cache Manager Routines</h2>
<p>Normally you would open files with Cache Manager enabled so let us look at its main routines that help us reduce I/O operations and enable our application to work faster:
1. Reading ahead
2. Lazy writing</p>
<h2 id="intelligent-read-ahead">(Intelligent) Read Ahead</h2>
<p>Cache Manager, keeps a history of the last two read request, for each file handle being used. With such history in mind it tries to determine the following patterns:</p>
<ol>
<li>User is doing sequential forward reads.</li>
<li>User is doing sequential backward reads.</li>
<li>User is doing reads in step sizes (offset 0 10 bytes, offset 1000 10 bytes, offset 2000 10 bytes&hellip;.)</li>
</ol>
<p>If a pattern corresponds with history read requests then the Cache Manager can anticipate which bytes you are going to read next and that is a big performance improvement.</p>
<h3 id="how-do-cache-flags-affect-read-ahead">How do cache flags affect Read Ahead?</h3>
<table>
<thead>
<tr>
<th>Flag</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>FileOptions.None</td>
<td>General (previously described) pattern detection algorithm is used when no cache flags are applied.</td>
</tr>
<tr>
<td>FileOptions.SequentialScan</td>
<td>Cache manager does not try to determine any patterns. More aggressive read ahead is used - two times as much data is prefetched than normally (Do not use it when doing backward reads).</td>
</tr>
<tr>
<td>FileOptions.RandomAccess</td>
<td>If you are constantly doing seeks, you are wasting time and memory pre-emptively reading data that your application will probably not need. This option disables read-ahead.</td>
</tr>
</tbody>
</table>
<h2 id="lazy-writing-write-back-cache">Lazy Writing (Write Back Cache)</h2>
<p>In order to flush as much data as possible to disk at once, write operations are piled up in memory until the decision is made by the Cache Manager. The decision about, which dirty pages should be flushed to disk and when, is made by the Cache Manager&rsquo;s lazy writer function that executes on a separate system worker thread once per second.</p>
<h3 id="how-do-cache-flags-affect-lazy-writing">How do cache flags affect Lazy Writing?</h3>
<table>
<thead>
<tr>
<th>Flag</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>FileAttributes.Temporary</td>
<td>(Disable lazy writing) File system tries to keep the data in memory and flushes it only in case of memory shortage. This attribute is commonly combined withFileOptions.DeleteOnClose. Both of these flags combined are ideal for small temporary files since there will most likely be no hits to the hard disk.</td>
</tr>
<tr>
<td>FileOptions.WriteThrough</td>
<td>Cache manager still caches data but only for file-read operations. Write changes are written to disk instantly. This is useful for reducing the risk for data loss. Warning! A lot of <a href="http://blogs.msdn.com/b/oldnewthing/archive/2014/03/07/10505524.aspx">resources</a> are stating that all SATA drives ignore this flag.</td>
</tr>
</tbody>
</table>
<h2 id="cache-usage-performance-counters">Cache Usage Performance Counters</h2>
<p>There are a lot of counters in the Cache section of Performance monitor that can help you evaluate cache usage. The two that relate to this blog post the most are Read Aheads/sec and Lazy Write Flushes/sec. The first one displays the rate at which the Cache Manager detects that a file is being accessed sequentially and the latter displays the rate of periodical flushes that are performed by the lazy writers thread. If you want to see all flushing activity use Data Flushes/sec counter.</p>
<h2 id="summary">Summary</h2>
<p>When you step out of your comfortable .NET bubble, wonderful things can happen. I discovered way more information when learning about native flags that I did with FileOptions Enumeration. If you, like myself, never did any native C++ Windows programming, reading about internal mechanisms like Cache Manager and how to control it really helps you to better understand and optimise your managed .NET application.</p></description>
</item>
<item>
<title>Keep your Mercurial revision history clutter free</title>
<link>http://www.ilovetodeletecode.com/post/2014/05/19/keep-your-mercurial-revision-history-clutter-free/</link>
<pubDate>Mon, 19 May 2014 16:10:04 +0000</pubDate>
<guid>http://www.ilovetodeletecode.com/post/2014/05/19/keep-your-mercurial-revision-history-clutter-free/</guid>
<description><p>I have been using Mercurial revision control system as a weapon of choice for quite some time now. For controlling changes made to source code, as for backing up to a private repository hosted at Bitbucket I used the basic commands for committing, adding, pushing and pulling files to/from the local /remote repository on a daily basis. And that was sufficient for my needs&hellip;.until recently. I stumbled upon an interesting blog post titled <a href="http://elegantcode.com/2014/02/15/segregate-your-code-commits-into-tiny-topical-changes/comment-page-1/">&ldquo;Segregate your commits into tiny topical changes&rdquo;</a> that made me think of how my current revision history on a project looks like&hellip;</p>
<p></p>
<figure >
<img src="http://www.ilovetodeletecode.com/img/Post3_history_example.png" />
</figure>
<p>This kind of spaghetti history is very difficult to review. The commits are not topical and there is a lot of cognitive noise just because of backup commits. And I surely wouldn’t want to share code from revision 4 when I send my changes upstream and share them with other developers. It would be nice if you could change history a bit before you push the delivered feature to the public repository, so it would look something like this:</p>
<figure >
<img src="http://www.ilovetodeletecode.com/img/Post3_better_history_example.png" />
</figure>
<h2 id="how-to-make-changesets-mutable-in-mercurial">How to make changesets mutable in Mercurial</h2>
<p><a href="https://www.mercurial-scm.org/wiki/MqExtension">Mercurial Queues</a> is an extension that rewrites history. Using MQ we can remove, reorder or fold (merge) committed changesets. This functionality does not come out of the box, first we must configure the extension which is being distributed along with Mercurial. In order to enable the MQ Extension, we modify the local .hgrc file. We can do this directly or through TortoiseHG repository settings dialog:</p>
<ol>
<li>modifying local .hgrc file
[extensions]
mq =
</li>
<li>Check TortoiseHG-&gt;Repository settings-&gt;Extensions-&gt;mq</li>
</ol>
<p>After the MQ Extension is enabled, Mercurial will include additional commands which we can print with hg help mq, but since this is a TortoiseHG tutorial, I will show you how to work with MQ through TortoiseHG Workbench.</p>
<h2 id="goals">Goals</h2>
<p>Using MQ Extensions we will modify our existing revision history, before we make it public. We will try to:</p>
<ol>
<li>Delete changeset 4 which was backed out.</li>
<li>Reorder changesets so that refactoring changeset 5 comes after feature A is implemented.</li>
<li>Split a changeset by file or by content using the Shelve tool. This is useful if one commit contains changes related to multiple topics. For example changeset 5 where we did some refactoring also contains logical changes that are part of the feature A implementation.</li>
<li>Fold (merge) all changesets that represent topical commit feature A into one changeset.</li>
</ol>
<p>Since we will modify the revision history directly, we cannot rollback if something goes wrong so it’s better to have a cloned repository for the sake of backup before we start.</p>
<p>To fully understand Mercurial Queues and the history behind patch management I also suggest reading <a href="http://hgbook.red-bean.com/read/managing-change-with-mercurial-queues.html">Chapter 12</a> from the official Mercurial guide. But just the introduction without code samples, because they are not up to date.</p>
<p>Long story short, patches deal with handling new code contributions from developers that do not have full commit access to a repository. This is useful on open source projects. But In this case we will use patches to prevent making permanent changes to our revision history and so making it more clutter free.</p>
<h2 id="treat-changesets-as-patches">Treat changesets as patches</h2>
<p>In order to manipulate changesets we must first add them to the Patch Queue. To view the Patch Queue window select &ldquo;View-&gt;Show Patch Queue&rdquo; from TortoiseHG Workbench:</p>
<figure >
<img src="http://www.ilovetodeletecode.com/img/Post3_WorkBench.png" />
</figure>
<p>If MQ was properly enabled we should see a &ldquo;Modify History&rdquo; menu item in the context menu of the selected changeset.</p>
<h2 id="goal-1-delete-changeset-4">Goal 1: Delete changeset 4</h2>
<p>Select changeset 4 and import it to MQ. After a successful import you should see a new patch was added to the queue named &ldquo;<revision>.diff&rdquo;. This patch was pushed on top of the repository stack and corresponds to the selected changeset:</p>
<figure >
<img src="http://www.ilovetodeletecode.com/img/Post3_delete_changeset.png" />
</figure>
<p>The next step is to unapply the patch. We can do this from the Patch Queue or from the context menu Changeset 4 &ldquo;Modify History-&gt;Unapply patch&rdquo;:</p>
<figure >
<img src="http://www.ilovetodeletecode.com/img/Post3_unapply_one_patch.png" />
</figure>
<p>When a patch is unapplied its corresponding commit is displayed at the top of the repository graph in a grayed out color:</p>
<figure >
<img src="http://www.ilovetodeletecode.com/img/Post3_grayed_out_color.png" />
</figure>
<p>Now we delete the patch and with this the changeset that corresponds to it:</p>
<figure >
<img src="http://www.ilovetodeletecode.com/img/Post3_delete_patch.png" />
</figure>
<h2 id="goal-2-reorder-changesets">Goal 2: Reorder changesets</h2>
<p>Let’s put refactoring changes in a changeset following the Feature A implementation changesets. Select changeset 1 and import it to MQ, then unapply all patches:</p>
<figure >
<img src="http://www.ilovetodeletecode.com/img/Post3_goal2_reorder.png" />
</figure>
<p>Use drag &amp; drop to change the order of changesets. Carry changeset 4 to the end of the queue. You can do this in the Patch Queue window or in the main history view:</p>
<figure >
<img src="http://www.ilovetodeletecode.com/img/Post3_drag_adn_drop.png" />
</figure>
<h2 id="goal-3-split-a-changeset">Goal 3: Split a changeset</h2>
<p>Using the shelve tool TortoiseHg &ldquo;Repository-&gt;Shelve&rdquo; we can move changes between unapplied patches. We can move all the changes made to a file or only chunks of changes inside a file. To move changes we select a target and destination patch in the left and right toolbar then we can use:</p>
<ol>
<li>The list view to select and move changes on a file basis.</li>
<li>The text editor to select and move chunks of changes.</li>
</ol>
<figure >
<img src="http://www.ilovetodeletecode.com/img/Post3_split.png" />
</figure>
<h2 id="goal-4-fold-merge-changesets">Goal 4: Fold (merge changesets)</h2>
<p>Re-apply patch 1 then select patches 2, 3, 5, 6, 7 and click &ldquo;Fold patches&rdquo;. This will merge selected patches into the topmost applied patch 1:</p>
<figure >
<img src="http://www.ilovetodeletecode.com/img/Post3_fold.png" />
</figure>
<p>A commit dialog will appear, containing comments from all the folded patches separated by a three asterisk syntax. Edit patch message and commit changes:</p>
<figure >
<img src="http://www.ilovetodeletecode.com/img/Post3_patch_fold.png" />
</figure>
<p>Now Re-apply patch 4:</p>
<figure >
<img src="http://www.ilovetodeletecode.com/img/Post3_reaply_patch4.png" />
</figure>
<p>Turn the applied patches back to a regular Mercurial changeset with the &ldquo;Finish Patch&rdquo; command:</p>
<figure >
<img src="http://www.ilovetodeletecode.com/img/Post3_finish_patch_command.png" />
</figure>
<p>Our changesets are now ready to be pushed to a public repository and shared with the other collaborators.</p></description>
</item>
<item>
<title>Refactoring a spaghetti C# serial communication app & asynchronous control flow modeling with async/await</title>
<link>http://www.ilovetodeletecode.com/post/2014/02/05/refactoring-a-spaghetti-c-serial-communication-app-asynchronous-control-flow-modeling-with-asyncawait/</link>
<pubDate>Wed, 05 Feb 2014 11:08:28 +0000</pubDate>
<guid>http://www.ilovetodeletecode.com/post/2014/02/05/refactoring-a-spaghetti-c-serial-communication-app-asynchronous-control-flow-modeling-with-asyncawait/</guid>
<description><p>A couple of months back I&rsquo;ve got a task of implementing a protocol for an embedded device sending/receiving streams of data through serial communication. Together with this task I also inherited a Windows Forms C# application that already did some of the previous but in a different protocol with a different device. The more I struggled to understand and reuse some of the existing infrastructure, the more I wanted to rewrite the whole thing (sounds familiar?).</p>
<p>Some facts about existing code:</p>
<ul>
<li>Each scenario was implemented as a complex state machine with long switch statements.</li>
<li>For each scenario a heavy weight enum with all possible states was defined.</li>
<li>State machine transitions were implemented with a table of Action delegates paired with already mentioned enumerations in a key/value dictionary.</li>
<li>Events were used to signal status of long running operations.</li>
</ul>
<p>Such code was really hard to read and even harder to maintain. And if you don’t have the documentation for the device&rsquo;s programming interface it&rsquo;s all even harder.</p>
<p>In this post I&rsquo;m trying to solve two things:</p>
<ol>
<li>How to design a more robust flexible issue/response system used for communicating with the serial device.</li>
<li>How to model complex asynchronous control flows, without blocking, using Task Parallel Library and C# 5.0 async/await.</li>
</ol>
<p></p>
<h2 id="use-the-state-pattern-to-get-rid-of-if-switch-statements">Use the state pattern to get rid of if/switch statements</h2>
<p>According to Gang of Four we should use the state pattern in either of the following cases:</p>
<ul>
<li>An object&rsquo;s behavior depends on its state, and it must change its behavior at run-time depending on that state.</li>
<li>Operations have large, multipart conditional statements that depend on the object&rsquo;s state. This state is usually represented by one or more enumerated constants. Often, several operations will contain this same conditional structure. The State pattern puts each branch of the conditional in a separate class. This lets you treat the object&rsquo;s state as an object in its own right that can vary independently from other objects.</li>
</ul>
<p>Using state pattern we can make a more object oriented state machine so that each behavior specific to one state is encapsulated into a separate object. Such separation of state specific logic simplifies our code as it reduces the need for large conditional statements and makes it flexible so that it is easy to add new states in the future.</p>
<h4 id="players-in-the-state-pattern">Players in the state pattern:</h4>
<table>
<thead>
<tr>
<th>Gang of Four</th>
<th>Our example</th>
</tr>
</thead>
<tbody>
<tr>
<td>Context</td>
<td>EmbeddedDevice</td>
</tr>
<tr>
<td>IState (defines an interface for encapsulating the behavior associated with a particular state of the Context)</td>
<td>ICommand</td>
</tr>
<tr>
<td>StateA : IState</td>
<td>OpenFileCommand : ICommand</td>
</tr>
</tbody>
</table>
public interface ICommand
{
IEnumerable<byte> CommandInstruction { get; }
void Proccess(IEnumerable<byte> data);
bool IsFinished { get; }
}
<h2 id="collaborations">Collaborations</h2>
<ol>
<li>Context delegates state-specific requests to the current ConcreteState object.
EmbeddedDevice type delegates serial data to command that is currently being executed. Delegation is performed via state&rsquo;s method Process (byte [] data) defined in ICommand.</li>
<li>Context is the primary interface for clients. Clients can configure a context with State objects. Once a context is configured, its clients don&rsquo;t have to deal with the State objects directly.
Clients issue new commands by adding it to the EmbeddedDevice command queue. Once the command is queued the client needs to wait to be signaled with the result.</li>
<li>Either Context or the ConcreteState subclasses can decide which state succeeds another and under what circumstances.
EmbeddedDevice checks the state of the current command with the IsFinished property defined in ICommand. If necessary a transition is made simply by switching the CurrentCommand variable to the next command from command queue and signaling to the client that the command has finished executing. When the next command starts executing the first thing that EmbeddedDevice does is write to serial port bytes from the CommandInstruction property.</li>
</ol>
<p>Using such partitioning adds another nice side effect. States are modeled to commands this means that logic specific to each command is implemented in its own class. This way only by looking the file structure of the project you can get familiar right away with the internal working of the device you are communicating with. Inspecting each command object instantly gives you information about how the command has to be issued and how the response for each command looks like.</p>
<figure >
<img src="http://www.ilovetodeletecode.com/img/Post2_structure.png" />
</figure>
<h2 id="achieve-readable-asynchronous-control-flow">Achieve readable asynchronous control flow</h2>
<p>The best way to represent future I/O bound work that yet has not finished is through the <a href="http://msdn.microsoft.com/en-us/library/dd460717(v=vs.110).aspx">TPL (Task Parallel Library)</a>. TPL is available since C# 4.0 and is replacing obsolete asynchronous patterns like <a href="http://msdn.microsoft.com/en-us/library/ms228963(v=vs.110).aspx">APM (Asynchronous Programming Model - BeginMethod, EndMethod, and IAsyncResult)</a> and <a href="http://msdn.microsoft.com/en-us/library/wewwczdw(v=vs.110).aspx">EAP (Event Based Asynchronous Pattern - signaling with events)</a>.</p>
<p>In the heart of TPL are the Task type and its generic subclass Task<TResult>. Task represents an asynchronous operation that does some initialization, maybe spawns a new thread (or maybe not) and then immediately returns without blocking the calling thread. Tasks in our case (serial communication) are all I/O bound, so there is just a lot of waiting (no new threads need to be created).
The second most important class in TPL is TaskCompletionSource. It enables us to take any operation (that doesn&rsquo;t yet follow TPL) and expose them as a Task. Exposing an asynchronous operation as a Task gives us:</p>
<ul>
<li>Monitor/callback mechanism.</li>
<li>Easily retrieve a return value.</li>
<li>Failure mechanism - exception propagation.</li>
<li>Composition - ability to chain Tasks together.</li>
<li>Separation of concerns - concurrency logic is not mixed with process logic.</li>
</ul>
<p>Lets see how we wrap TaskCompletionSource around our external asynchronous operation. We add the command to the execution queue together with the corresponding TaskCompletionSource object. TaskCompletionSource exposes a Task object that is completely controlled by its parent. We return this Task to the caller. Providing a TaskCompletionSource for each command through the process of execution gives us a mechanism to signal back to the caller the state of the ongoing command execution (SetCanceled, SetException, SetResult). Returning a Task from the AddToExecutionQueue function enables the caller to monitor that state and utilize a continuation (callback) that executes after completion.</p>
public Task<TReturn> AddToExecutionQueue<TReturn>(TReturn command)
{
var tcs = new TaskCompletionSource<TReturn>();
_commandQueue.Enqueue(new CommandTaskPair(command,tsc));
return tcs.Task;
}
<h2 id="structuring-asynchronous-flow-of-control">Structuring asynchronous flow of control</h2>
<p>Lets look at a more complex asynchronous workflow example. The process of downloading a file from the device is a nice example to start with. It consists of retrieving file information, reading error handling and then cleaning up the resources at the end. These are all long-running processes that together form a more complex asynchronous flow of control which we&rsquo;ll try to model using 3 different approaches:</p>
<ol>
<li>Nesting continuations using lambda expressions.</li>
<li>Writing separate functions.</li>
<li>Using C# 5 language constructs async/await.</li>
</ol>
<figure >
<img src="http://www.ilovetodeletecode.com/img/Post2_workflow.png" />
</figure>
<h2 id="approach-1-nesting-continuations-using-lambda-expressions">Approach 1: Nesting continuations using lambda expressions</h2>
<p>Each next step needs to be invoked from the previously completed one. Such complex sequence of steps result in nested spaghetti continuations which are hard to follow and maintain. Next problem is that we are downloading file chunk by chunk so we need a looping construct, but looping constructs don&rsquo;t go hand in hand with continuations. Because the next iteration should be triggered from the continuation itself we would have to use recursion. The only advantage with nested lambdas are closures which allow you to pass local state to the function without the need for parameters, but beware of closures if you don’t understand them fully, because funny things can happen :).</p>
public Task<long> Download()
{
long bytesTransfered = 0;
const int blockSize = 32 * 1024;
var tcs = new TaskCompletionSource<long>();
var openFileCmd = new OpenFileCmd(_fileInfo.Name, FileAccess.Read);
Task<OpenFileCmd> fileHandleTask = _embeddedDevice.AddToExecutionQueue(openFileCmd);
fileHandleTask.ContinueWith(fileHandleFinished =>
{
int fileHandle = fileHandleFinished.Result.SuccessResult;
var readBlockCmd = new ReadBlockCmd(fileHandle, blockSize);
Task<ReadBlockCmd> readBlockTask = _embeddedDevice.AddToExecutionQueue(readBlockCmd);
Action<Task<ReadBlockCmd>> recursiveDownload = null;
recursiveDownload = readBlockFInished =>
{
byte[] bytesRead = readBlockFInished.Result.SuccessResult.ActualData.ToArray();
bytesTransfered += bytesRead.Count();
Proccess(bytesRead);
if (bytesTransfered < _fileInfo.Length)
{
readBlockCmd = new ReadBlockCmd(fileHandle, blockSize);
Task<ReadBlockCmd> nextBlock = _embeddedDevice.AddToExecutionQueue(readBlockCmd);
nextBlock.ContinueWith(recursiveDownload);
}
else
{
var closeFileCmd = new CloseFileCmd(fileHandle);
Task<CloseFileCmd> closeHandle = _embeddedDevice.AddToExecutionQueue(closeFileCmd);
closeHandle.ContinueWith(closedHandle => tcs.SetResult(bytesTransfered));
}
};
readBlockTask.ContinueWith(recursiveDownload);
});
return tcs.Task;
}
<h2 id="approach-2-writing-separate-functions">Approach 2: Writing separate functions</h2>
<p>Splitting callbacks into separate functions gives us a more readable control flow and also an easier way to achieve looping behavior. One drawback is that we have to pass parameters or use member variables in order to preserve context. Compared with lambda expressions, we are better off splitting functions but at the end of the day nothing can bring us a more natural control flow than approach 3.</p>
public class DownloadFile
{
private readonly EmbeddedDevice _embeddedDevice;
private DeviceFileInfo _fileInfo;
private TaskCompletionSource<long> _tcs;
long _bytesTransfered;
private int _fileHandle;
const int BlockSize = 32 * 1024;
public DownloadFile(EmbeddedDevice embeddedDevice)
{
_embeddedDevice = embeddedDevice;
}
public Task Download(DeviceFileInfo fileInfo)
{
_tcs = new TaskCompletionSource<long>();
_bytesTransfered = 0;
_fileInfo = fileInfo;
_fileHandle = 0;
var openFileCmd = new OpenFileCmd(_fileInfo.Name, FileAccess.Read);
Task<OpenFileCmd> fileHandleTask = _embeddedDevice.AddToExecutionQueue(openFileCmd);
fileHandleTask.ContinueWith(OnFilehandleReceived);
return _tcs.Task;
}
private void OnFilehandleReceived(Task<OpenFileCmd> fileHandleTask)
{
_fileHandle = fileHandleTask.Result.SuccessResult;
var readBlockCmd = new ReadBlockCmd(_fileHandle, BlockSize);
Task<ReadBlockCmd> firstBlock = _embeddedDevice.AddToExecutionQueue(readBlockCmd);
firstBlock.ContinueWith(OnBlockReceived);
}
private void OnBlockReceived(Task<ReadBlockCmd> readBlockTask)
{
IEnumerable<byte> data = readBlockTask.Result.SuccessResult.ActualData;
ProccessData(data);
_bytesTransfered += data.Count();
if (_bytesTransfered < _fileInfo.Length)
{
var readBlockCmd = new ReadBlockCmd(_fileHandle, BlockSize);
Task<ReadBlockCmd> nextBlock = _embeddedDevice.AddToExecutionQueue(readBlockCmd);
nextBlock.ContinueWith(OnBlockReceived);
}
else
{
var closeFileCmd = new CloseFileCmd(_fileHandle);
Task<CloseFileCmd> closeHandle = _embeddedDevice.AddToExecutionQueue(closeFileCmd);
closeHandle.ContinueWith(OnResourcesCleaned);
}
}
private void OnResourcesCleaned(Task<CloseFileCmd> closedHandle)
{
_tcs.SetResult(_bytesTransfered);
}
}
<h2 id="approach-3-using-c-5-0-language-constructs-async-await">Approach 3: Using C# 5.0 language constructs async/await</h2>
<p>Async/await enables us to program asynchronous workflows in a linear way, that is, we can write our program as we would synchronously but without tying up a thread. If a method, called with await, has not yet completed, the execution immediately returns to the caller. When the method completes, the execution jumps back to the method where it left of, preserving all local state. What compiler does under the covers when it stumbles upon an await operator is very similar to yield return in an iterator, that is, a lot of boiler plate code gets generated in order to achieve pause/resume functionality using state machines.</p>
public async Task<long> Download(DeviceFileInfo fileInfo;)
{
long bytesTransfered = 0;
const int blockSize = 32 * 1024;
var openFileCmd = new OpenFileCmd(fileInfo.Name, FileAccess.Read);
OpenFileCmd fileHandleTask = await _embeddedDevice.AddToExecutionQueue(openFileCmd);
int fileHandle = fileHandleTask.SuccessResult;
while (bytesTransfered < fileInfo.Length)
{
var readBlockCmd = new ReadBlockCmd(fileHandle, blockSize);
ReadBlockCmd readBlockTask = await _embeddedDevice.AddToExecutionQueue(readBlockCmd);
byte[] bytesRead = readBlockTask.SuccessResult.ActualData.ToArray();
Proccess(bytesRead);
bytesTransfered += bytesRead.Count();
}
var closeFileCmd = new CloseFileCmd(fileHandle);
await _embeddedDevice.AddToExecutionQueue(closeFileCmd);
return bytesTransfered;
}
<p>In summary, hiding callbacks with async/await language asynchrony support in C# 5.0 enables us to write code like we are used to and prevents complex asynchronous workflows (completion of multiple tasks, error handling…) from becoming unreadable and difficult to maintain. Even when targeting .NET 4 there is a <a href="https://www.nuget.org/packages/Microsoft.Bcl.Async/1.0.16">package</a> on NuGet Microsoft.Bcl.Async which enables async/await keywords and includes the new Task based extension methods.</p></description>
</item>
<item>
<title>Prevent large file commits with PowerShell and Mercurial hooks</title>
<link>http://www.ilovetodeletecode.com/post/2013/10/31/prevent-large-file-commits-with-powershell-and-mercurial-hooks/</link>
<pubDate>Thu, 31 Oct 2013 12:53:22 +0000</pubDate>
<guid>http://www.ilovetodeletecode.com/post/2013/10/31/prevent-large-file-commits-with-powershell-and-mercurial-hooks/</guid>
<description><p>Because of Mercurial&rsquo;s immutable historical record of project files there is no way to make a committed file completely disappear from the history tree. Our only hope in this case it&rsquo;s not Obi-Wan but the hg rollback command if we act quickly. Since this command only works for one most recent operation <a href="http://stackoverflow.com/questions/3288865/how-to-keep-a-mercurial-repository-small/">(How to keep a Mercurial repository small?)</a> it will not work if we already committed one or more changes after the change that we would like to completely remove <a href="http://stackoverflow.com/questions/8466669/accidentally-committed-a-large-amount-of-raw-data-in-mercurial-how-do-i-keep-it">(Accidentally committed a large amount of raw data in Mercurial, how do I keep it from overloading my Bitbucket repository?)</a>. Our problems escalate even faster if we already pushed or pulled changes to a remote repository and our bad commit propagated to other collaborators.</p>
<p>Two most common cases of brown paper bag commits <a href="http://hgbook.red-bean.com/read/finding-and-fixing-mistakes.html#sec:undo:aaaiiieee">(a commit that is so bad you want to pull a brown paper bag over your head)</a> are accidentally committing sensitive information files (passwords, connection strings&hellip;) and large files. The latter case is less critical because it only increase s repository size and the time it takes to perform operations like pull or clone.</p>
<p>If you are interested in how to prevent yourself from committing large files and are more comfortable with PowerShell then please read on. If you are more of an indentation than squirrely brackets guy then you can check this similar solution in Python - <a href="http://stackoverflow.com/questions/2551719/mercurial-hook-to-disallow-committing-large-binary-files">Mercurial hook to disallow committing large binary files</a>.</p>
<p></p>
<h2 id="workflow">Workflow</h2>
<p>Our solution has a simple 3 step workflow</p>
<ol>
<li>On each commit retrieve files that have been added to the repo in current change set.</li>
<li>Get the number of files that have size greater than maximum allowed size..</li>
<li>Roll back or complete commit transaction based on the information from step 2.</li>
</ol>
<h2 id="controlling-repository-events-with-hooks">Controlling repository events with hooks</h2>
<p>Triggers that allow you to perform custom actions in response to events that occur on a Mercurial repository are called hooks. Because we need to execute a custom script every time a commit occurs there are two hooks that might interest us.</p>
<ol>
<li>Commit - runs after a new changeset has been created in the local repository.</li>
<li>Pretxncommit - runs after a new changeset has been created in the local repository, but before the transaction is completed.</li>
</ol>
<p>For obvious reasons we will go with pretxncommit. This hook will not only call our custom PowerShell script every time a commit, but it will also pass parameters in form of environmental variables to it. Each Hook parameter gets prefixed with &ldquo;HG_”. We can list all parameters that are available from Mercurial context applying a filter to PowerShell &ldquo;Env&rdquo; drive&rsquo;s content:</p>
dir -path:"env:" | Where-Object {$_.Name -like "HG_*"} | Select -Property Name,Value
<p>Since pretxncommit is a controlling hook we can control the success of the operation using exit codes from our PowerShell script. Return a non-zero to roll back the transaction or 0 to complete it. We will exit our PowerShell script using the number of files greater than maximum size allowed.</p>
<h2 id="defining-the-pretxncommit-hook-as-an-external-program">Defining the pretxncommit Hook as an external program</h2>
<p>There are two ways in which a Hook gets executed. In our case the execution is &ldquo;shell out&rdquo; to another process. An alternative would be a Python function that is executed within the Mercurial process. In-process hooks are faster and have complete access to Mercurial API but you have to write code in Python.</p>
<p>Hooks are defined in the [hooks] section of your local (hg\hgrc) or global (%USERPROFILE%\mercurial.ini) settings file. Each hook is defined in a type, name and action like format. You can edit settings file directly or use TortoiseHg settings dialog:</p>
[hooks]
pretxncommit.fileSizeCheckHook = powershell.exe -File "C:\Users\Jernej\Documents\WindowsPowerShell\pretxncommit.ps1"
<figure >