Updates to V&V lectures, addition of description versus specification, PoC details

Lectures/L09_VerificationAndValidation/VerificAndValid.tex

@@ -79,7 +79,7 @@
 \item Enough issues, but shallow 1/2
 \item Enough issues and deep (not surface) 2/2
 \ei
-\item Issues are due 2 days after your colleague assigns your issue
+\item Issues are due 2 days after being assigned
 \ei
 \item Reading week next week, no 741 classes
 \item Work on your VnV plan during reading week

Lectures/L10_VnVContinued/VnVContinued.tex

@@ -46,7 +46,7 @@
 \item Start recording
 \item Administrative details
 \item Questions?
-\item Finish what started last day
+%\item Finish what started last day
 \item Nonfunctional software testing
 \item Theoretical foundations of testing
 \item Complete coverage principle
@@ -64,24 +64,37 @@
 \frametitle{Administrative Details}
 
 \bi
-\item As the GitHub repo owner
+\item Drasil projects look at \href{http://learnyouahaskell.com/} {Learn you a
+    Haskell for Great Good} \ei
+
+\end{frame}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\begin{frame}
+\frametitle{Admin Details: VnV Presentations}
+
 \bi
-\item Add your reviewers as collaborators
-\item When your project is ready for review
+\item Not everyone will do VnV presentations
+\item Select 1 or 2 of the following:
 \bi
-\item Assign your reviewers an issue for
-  them to create issues
-\item Assign the instructor to review
-\ei
+\item \textbf{Specific} functional system test cases
+\item \textbf{Specific} nonfunctional system test cases, such as
+\bi
+\item Performance profile
+\item Usability testing
 \ei
-\item As a GitHub reviewer
+\item SRS verification plan
+\item Automated testing and verification tools
 \bi
-\item Assigned 2 colleagues (see \texttt{Repos.xlsx} in repo)
-\item Provide at least 5 issues on their SRS
+\item Profiling tools
+\item Continuous integration
+\item Code coverage
+\item Linters
 \ei
-\item Reading week, no 741 classes
-\item V\&V template in repo
-\item Adding a V\&V checklist to repo
+\ei
+\item We would like a variety of topics presented
+\item If you are uncertain of your specific presentation plan, please ask
 \ei
 
 \end{frame}
@@ -89,42 +102,45 @@
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
 \begin{frame}
-\frametitle{Administrative Details: Report Deadlines}
-~\newline
-\begin{tabular}{l l l}
-System VnV Plan & Week 08 & Oct 28\\
-MG + MIS & Week 10 & Nov 25\\
-Final Documentation & Week 14 & Dec 9\\
-\end {tabular}
+  \frametitle{Admin Details: Proof of Concept Presentations}
 
 \bi
-\item The written deliverables will be graded based on the repo contents as of
-11:59 pm of the due date
-\item If you need an extension, please ask
-\item Two days after each major deliverable, your GitHub issues will be due
-\item Domain expert code due 1 week after MIS deadline
+\item Deepen your understanding by jumping into implementation
+\item Identify a risk with your code and implement enough to show you can
+  resolve it
+\item Looking for an actual demo with running code
+\item Presentation
+\bi
+\item Explicitly identify your risk
+\item Run your code
+\item Discuss your implementation
+\ei
+\item Simplify as much as necessary
+\item Do not use this code in your actual implementation
 \ei
 
 \end{frame}
 
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
 \begin{frame}
-\frametitle{Administrative Details: Presentations}
-
+\frametitle{Administrative Details: Report Deadlines}
 ~\newline
-\begin{tabular}{l l l}
-Syst.\ VnV Present & Week 07 & Week of Oct 21\\
-MG + MIS Syntax Present & Week 9 & Week of Nov 4\\
-MIS Semantics Present & Week 11 & Week of Nov 18\\
-Unit VnV or Impl.\ Present & Week 12/13 & Week of Nov 28\\
+\begin{tabular}{l l}
+%\textbf{SRS} & Oct 8\\
+\textbf{System VnV Plan} & Oct 29\\
+MG + MIS (Traditional) & Nov 19\\
+Drasil Code and Report (Drasil) & Nov 19\\
+Final Documentation & Dec 9\\
 \end {tabular}
 
 \bi
-\item Informal presentations with the goal of improving everyone's written
-  deliverables
-\item Domain experts and secondary reviewers (and others) will ask questions
-  (listed in Repos.xlsx file)
+\item The written deliverables will be graded based on the repo contents as of
+11:59 pm of the due date
+\item If you need an extension for a written deliverable, please ask
+\item You should inform your primary and secondary reviewers of the extension
+\item Two days after each major deliverable, your GitHub issues will be due
+%\item Domain expert code due 1 week after MIS deadline
 \ei
 
 \end{frame}
@@ -132,40 +148,61 @@ Unit VnV or Impl.\ Present & Week 12/13 & Week of Nov 28\\
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
 \begin{frame}
-\frametitle{Administrative Details: Presentation Schedule}
+\frametitle{Admin Details: Presentation Schedule}
 
 \bi
-\item Syst V\&V Plan Present
+\item Syst V\&V Plan Present (15 min)
 \bi
-\item \textbf{Monday: Deema, Peter}
-\item \textbf{Thursday: Sharon, Ao}
+\item Thurs, Oct 22: Ting-Yu, Mohamed, Naveen, Liz, Salah
 \ei
-\item MG + MIS Syntax Present
+\item Proof of Concept Demonstrations (15 min)
 \bi
-\item Monday:  Deema, Bo
-\item Thursday: Sasha
+\item Mon, Oct 26: Tiago, Mohamed, Xuanming, Parsa, Gaby
+\item Mon, Nov 2: Sid, Shayan, Leila, Xingzhi, Liz
+\item Thurs, Nov 12: Salah, John
 \ei
-\item MIS Syntax + Semantics Present
+\item MG Present (10 minutes)
 \bi
-\item Monday: Zhi, Peter
-\item Thursday:  Sharon, Ao
+\item Thurs, Nov 12: John, Tiago, Leila, Xuanming, Andrea
 \ei
-\item Unit VnV Plan or Impl.\ Present
+\item MIS Present
 \bi
-\item Monday: Bo, Sasha
-\item Thursday: Zhi, Peter, Ao
+\item Mon, Nov 16: Shayan, Parsa, Gaby, Sid, Xingzhi
+\ei
+\item Drasil Project Present (20 min each)
+\bi
+\item Thurs, Nov 26: Andrea, Naveen, Ting-Yu
 \ei
 \ei
 
 \end{frame}
 
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\begin{frame}
+\frametitle{Presentation Schedule Continued}
+
+\bi
+\item Test or Impl.\ Present (15 min each)
+\bi
+\item Mon, Nov 30: John, Salah, Liz, Xingzhi, Leila
+\item Thurs, Dec 3: Shayan, Naveen, Sid, Gaby, Seyed
+\item Mon, Dec 7: Ting-Yu, Xuanming, Mohamed, Andrea, Tiago
+\ei
+\item 4 presentations each
+\item If you will miss a presentation, please trade with someone else
+\ei
+
+\end{frame}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
 \begin{frame}
 \frametitle{Questions?}
 \begin{itemize}
-\item Questions about SRS?
+%\item Questions about SRS?
 \item Questions about V\&V?
+\item Questions about PoC?
 \end{itemize}
 \end{frame}
 
@@ -272,6 +309,30 @@ Unit VnV or Impl.\ Present & Week 12/13 & Week of Nov 28\\
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
+\begin{frame}
+\frametitle{Description Rather Than Specification}
+\begin{itemize}
+\item Test cases are often phrased as Expected = Calculated
+\item In scientific software you generally should not test for equality 
+\bi
+\item Absolute error within tolerance
+\item Relative error within tolerance
+\item If comparing matrices or vectors, consider using norms of residual
+\ei
+\item Even a specific tolerance often doesn't make sense in a scientific context
+\item Often your plan should be to \textbf{describe} the error rather than
+  \textbf{prescribe}
+\bi
+\item Plot of error versus problem size, or condition number, or ...
+\item Consider summarizing multiple tests with the infinity norm of the relative
+  error (or similar)
+\ei
+\item Your description plan is part of your V\&V plan!
+\end{itemize}
+\end{frame}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
 \begin{frame}
 \frametitle{White Box Versus Black Box Testing}
 \begin{itemize}