A standard library needs to be very robust. We don't want any surprises. The best way to avoid surprises is to have extremely good test coverage.
Add test coverage to the Menai tools
The best way to check if we're doing the right things in the library is to ensure we can reach and reliably execute every instruction of generated code. To do this I added --coverage to all the tools. I also make width clipping optional rather than on by default so we don't lose information.
For menai-test runs the coverage metrics are generated across all the tests in a test suite.
This give a result something like this:
──────────────────────────────────────────────────────────────────────
FUNCTION COVERAGE — base64-decode.menai
──────────────────────────────────────────────────────────────────────
Functions: 12
Fully exercised: 6
Complete but for dead bytecode: 0
With reachable gaps: 6
Instructions: 292
Executed: 250 85.62%
Reachable: 292 85.62%
Unreachable: 0
Executed Total Coverage Calls Status Function
---------- ---------- ---------- ---------- ------------------ ----------------------------------------
0 2 0.00% 0 gap <lambda> [line 157]
41 59 69.49% 4 gap decode-unpadded-bytes [line 184]
21 30 70.00% 3 gap decode-final-two [line 87]
34 40 85.00% 21 gap decode-group [line 67]
38 44 86.36% 18 gap decode-final-group [line 117]
46 47 97.87% 30 gap decode [line 217]
35 35 100.00% 24 full decode-padded [line 133]
2 2 100.00% 2 full <lambda> [line 157]
2 2 100.00% 18 full <lambda> [line 157]
2 2 100.00% 26 full <lambda> [line 157]
2 2 100.00% 15 full <lambda> [line 157]
27 27 100.00% 18 full decode-padded-bytes [line 165]
Reachable instructions not executed
──────────────────────────────────────────────────────────────────────
<lambda> [line 157] 0-1
decode-unpadded-bytes [line 184] 7-18, 53-58
decode-final-two [line 87] 21-29
decode-group [line 67] 28-30, 34-36
decode-final-group [line 117] 38-43
decode [line 217] 45 RAISE_ERROR 'No patterns matched in match expression'
Dead bytecode (unreachable from entry)
──────────────────────────────────────────────────────────────────────
NoneNote it's specifically looking for code that can never be reached as such code would be a compiler optimization error.
We can also get the cumulative numbers for an full test run:
Total: 1056/1056 passed
──────────────────────────────────────────────────────────────────────
CUMULATIVE FUNCTION COVERAGE
──────────────────────────────────────────────────────────────────────
Source files: 34
Functions: 422
Fully exercised: 369
Complete but for dead bytecode: 0
With reachable gaps: 53
Instructions: 13,540
Executed: 13,312 98.32%
Reachable: 13,540 98.32%
Unreachable: 0
Module Functions Full Gaps Coverage
-----------------------------------------------------------
base64-decode 12 6 6 85.62%
zip-create 5 2 3 91.28%
tar-create 17 14 3 93.01%
tar-extract 8 7 1 94.49%
gzip-decompress 3 1 2 95.80%
png-decode 10 6 4 95.94%
png-encode 4 1 3 96.35%
zip-entries 7 4 3 96.35%
bmp-decode 4 3 1 96.61%
deflate-compress 47 40 7 96.74%
xlsx-decode 16 13 3 96.88%
base64-encode 2 1 1 97.39%
bmp-encode 5 2 3 97.45%
zip-extract 9 8 1 97.59%
pptx-decode 16 14 2 97.91%
zlib-decompress 2 1 1 98.39%
html-encode 8 6 2 99.01%
stats 65 60 5 99.28%
csv-decode 5 4 1 99.47%
regexp 31 30 1 99.91%
csv-encode 3 3 0 100.00%
deflate-decompress 19 19 0 100.00%
docx-decode 6 6 0 100.00%
docx-encode 12 12 0 100.00%
gzip-compress 1 1 0 100.00%
html-decode 21 21 0 100.00%
json-decode 7 7 0 100.00%
json-encode 7 7 0 100.00%
pptx-encode 22 22 0 100.00%
tar-entries 10 10 0 100.00%
xlsx-encode 18 18 0 100.00%
xml-decode 11 11 0 100.00%
xml-encode 8 8 0 100.00%
zlib-compress 1 1 0 100.00%
Functions with reachable gaps (worst 20 of 53)
──────────────────────────────────────────────────────────────────────
base64-decode <lambda> [line 157] 0-1
base64-decode decode-unpadded-bytes [line 184] 7-18, 53-58
tar-extract extract-entry [line 90] 13-19
base64-decode decode-final-two [line 87] 21-29
png-decode channels-for [line 75] 8-12
tar-create split-path [line 62] 21-25, 29-33
deflate-compress find-match [line 252] 35-39, 46-50, 57-60
zip-create build-local-entry [line 59] 8-9, 22-32, 70
deflate-compress write-stored-block [line 478] 1-2, 20, 33-38
zip-entries read-eocd [line 34] 21-25
pptx-decode sld-ids [line 172] 11-14
xlsx-decode sheet-elements [line 185] 11-14
base64-decode decode-group [line 67] 28-30, 34-36
xlsx-decode shared-strings-of [line 246] 17-20
gzip-decompress skip-cstring [line 55] 10, 13
base64-decode decode-final-group [line 117] 38-43
deflate-compress encode-auto [line 766] 19-24, 43, 60
deflate-compress encode-block [line 755] 47-52, 58
tar-create build-entries [line 210] 14-15, 22-23
zip-extract entry-content [line 33] 25-28At the time of writing this, the test coverage is already at 100% on quite a large number of modules! A few have highlighted an interesting compiler question that I need to think about, but there's also now a test that pins the test coverage at 100% for some modules, and once I can do that for all modules then we'll pin this as a requirement for any new ones too.