Thoughts about cost function, "testing"

[audreyseo]

Here's one of the log files: latest.log

If you look to see where all of the costs are, there's a lot of them that have the minimum cost for whatever block we're doing at the moment, so the one that ends up being first is pretty arbitrary. I'm not sure if there's a way to make the cost function more fine-tuned to what we want without "over-fitting" our solution, per se.

We've been wondering what we should do if the behavior of the resulting program is not the same as the desired behavior, and I feel like it could do one of multiple things:

• it could simply interchange one of the parts with one of the results from the synthesizer that also has the minimum cost, and brute-force try every single combination
• it could try to modify the original ROS1 code until it changes something in the same place as the change in the output for the other program, which at least will pin-point some block that we could try a different result for
• using a diff between the synthesized run and the actual output, we could calculate some kind of "similarity index" between the two, and if it interchanges solution 1 for solution 2 for block A, and that makes it more "similar", then it should "keep" solution 2 and move on to looking at block B's solutions and changing those out.

However, this isn't going to happen until I fix #42.