Recomendações para prevenir o COVID-19

FAVOR DIVULGAR

Python3 - String data as csv (file or list) objects

I spent the last three days reading Python guides, manuals, blogs, tutorials, e-books, etc. just to be able to start working with data in text format I gathered from sensors and text format tables.

I got stuck because I was incorrectly trying to use csv methods to manipulate a file that I read into a string, and wrongly though I could just format it as a standard csv file layout.  This just does not work!

The official doc for csv.reader() at https://docs.python.org/2/library/csv.html is very helpful, which stats this:

    "file objects and list objects are both suitable"

The solution is to convert your formatted string to a "file or list object", (what ?!?)

The side effect of not doing this is that all csv methods don't fetch the items correctly and ignores your delimiters (ie. like the comma)

So you have two ways of doing this:

First:  you can import StringIO to perform this conversion:

   from IO import StringIO
   import csv

After that you need to code the conversion, and for this example lets assume your string is named: target

   new_csv = StringIO(target)

But wait!!! Hold your horses!!! Keep on reading.

Second: Or choose a easier way, that is: make a list from your string:

   new_csv = target.splitlines()

You can check the type of objects with:

   print('type of target  = '+str(type(target)))
   print('type of new_csv = '+str(type(new_csv)))

Now you are ready to use all the csv methods to analise/manipulate your tables/data

Cheers!

Check your link speed to all servers in location

So I had same problems with my local xDSL provider.

To be able to test the speed of connection we can use the speedtest site or a bash script from a GitHub developer, but I created the following bash script which will use the speedtest script to test the speed on all the servers in my city and save it to a log file.

#!/bin/bash
#
# speedtest all my city servers
#
# Author: braselectron.com 05/DEZ/2019
#
#
# ref: 
# curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -
#
# dependencies: 
#
#  1) wget https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py
#
#  2) ubuntu 18.04
#
#  2.1) GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)
#
#  3) http://ipinfo.io/ip   # for external IP fetch
#
#
targetcity="Miami"
# if target city name has spaces change it for grep \s like 
# this: "Salt\sLake\sCity"
gtarget=$( echo "$targetcity" | sed "s/ /\\\s/g )
#
servers=$( python speedtest.py --list | grep $gtarget )
# debug:
# echo -e "I found these servers:\n$servers"
servnum=$( echo -e "$servers" | cut -d ')' -f 1 )
# debug:
# echo -e "These are the server numbers:\n$servnum"
tot=$( echo -e "$servnum" | wc -l )
echo -e "\nRunning, please wait $tot servers to go:\n"
#
echo -e "Speedtest "$( wget -qO - http://ipinfo.io/ip )" "$( date +%F )"\n" > speedtest_$( date +%F ).log
#
i=1
#
for serv in $servnum
   do
     echo -e "$i of $tot - ($serv): "
     python speedtest.py --server $serv 2>&1 | grep ':' |\
     sed "s/$targetcity/$serv/g" |\
     sed "s/.*ERROR.*/Server is not responding/g" |\
     tee -a speedtest_$( date +%F ).log
     echo -e "" | tee -a speedtest_$( date +%F ).log
     sleep 1
     ((i++))
     #
   done
echo ""

Cheers!

How to list, delete and include iptables rules.

First you should list all rules and index tem so you can pinpont your request correctly:

 sudo iptables -nL --line-numbers

Note: the -n means to not use DNS to resolve IP; the -L means to list rules; --line-numbers you can guess what it means.

Now you can choose a "chain", in this example: "INPUT" and use the "-D" to indicate delete and the final number is point to rule "5"

 sudo iptables -D INPUT 5

You can also add a rule, and bellow I will use chain INPUT for a known hacker from Kenya, Africa:

 sudo iptables -I INPUT 1 -s 197.248.0.0/16 -j DROP

That's it for now. 
Cheers!  and Shields up!

How to recover packages to local cache - Ubuntu

A few days ago I had to "sudo apt clean" my local cache to uninstall a bad package, but I needed to recover my cache, with the installed packages, since I work often in offline places, so that I can recover from a broken system without an active Internet connection.

So to be able to recover, or better, rebuild (or re-cache) your packages to your local storage (ie HDD or SSD) that usually is located at: /var/cache/apt/archives  we have to do this:

1) get the installed package names:

dpkg -l | grep "^ii" | awk ' {print $2} ' > pkg_names.txt

2) get the installed package versions:

dpkg -l | grep "^ii" | awk ' {print $3} ' > pkg_version.txt

3) combine all package names and versions:

paste -d"=" pkg_names.txt pkg_version.txt > pkg_combine.txt

4) now download packages to cache:

cat pkg_combine.txt | xargs sudo apt-get -y install --reinstall --download-only

REF:

dpkg -l | grep "^ii"| awk ' {print $2} ' | xargs sudo apt-get -y install --reinstall --download-only

dpkg -s firefox | grep Version | awk ' {print $2} '

sudo apt-get install package=version

Todo:

Timeout each line to avoid long loops (when source is dead or pkg not available at repository)

List total hits of iptable drops

I have been busy making and studying defense strategies to better implement and monitor my firewall system.

So after you have decided to insert DROP rules in your iptables, it is a good practice to check the statistics of how efficient the additional "load" has been to your overall performance, that is, is it effective or should you leave it for fail2ban to control the hits ?

I made a bash script to summaries the hits on iptables rules to avoid all the stdout that obfuscates the  important information.

As usually sad, use at your own risk.  Make a backup plan before proceeding.

#!/bin/bash
#
# check how many hits on iptables drop rule
# by braselectron.com  OCT 16, 2019
#
# iptables formated output example needed:
# '0 0 DROP all -- any any 47.203.94.77 anywhere'
#
# so now clear and fix the spaces on the output
#
readarray iptls <<< "$(sudo iptables -vnL | grep DROP |\
 sed 's/  / /g' | sed 's/  / /g' | sed 's/  / /g' |\
 sed 's/^ //g')"
#
# debug point
# echo "iptls lenght is ${#iptls[@]}"
#
echo -e "Hits\tTarget Denied"
for element in "${iptls[@]}"
do
  #
  # debug point
  # echo "> $element"
  #
  verify=( $(echo "$element" | cut -d " " -f 1) )
  #
  # debug point
  # echo "verify = $verify"
  #
  if [ "$verify" -ne "0" ]; then
     hits="$verify"
     target=( $(echo "$element" | cut -d " " -f 8) )
     echo -e "$hits\t$target"
  fi
done

If all goes well you will get a output similar to this:

Hits    Target Denied
1       198.108.66.0/23
4       92.118.161.0/24
3       92.118.160.0/24
1       74.82.47.0/24
1       185.173.35.0/24
1       71.6.128.0/17
2       122.228.0.0/16
4       95.154.101.209
1131    139.59.13.150

But this is based on my active iptables rules.

Shields up captain!
Cheers!

How to fix id3tag of mp3 files

Preliminary notes:

This post is the sum of many hours of researching the web for solutions, reading for many hours: man, info and community pages. And a lot of testing.

This is a solution for Linux users (probably also good for MacOS too).  MS Windows user, sorry, you need to contact MS support for help.

Part of the following is cut/past form other sources I found on the web.  Thank you for the active community of Linux users like me.

As usually noted, the following is for sharing knowledge only, do it at your own risk!  Make a backup plan before proceeding.

Basic knowledge:

Libavformat (lavf) is a library for dealing with various media container formats.  Its main two purposes are demuxing - i.e. splitting a media file into component streams, and the reverse process of muxing - writing supplied data in a specified container format.

The MP3 muxer writes a raw MP3 stream with the following optional features:

An ID3v2 metadata header at the beginning (enabled by default). Versions 2.3 and 2.4 are supported, the id3v2_version private option controls which one is used (3 or 4).

Setting id3v2_version to "0" disables the ID3v2 header completely.

The muxer usually support writing attached pictures (APIC frames) to the ID3v2 header. The pictures are supplied to the muxer in form of a video stream with a single packet. There can be any number of those streams, each will correspond to a single APIC frame. The stream metadata tags title and comment map to APIC description and picture type respectively.

See http://id3.org/id3v2.4.0-frames for allowed picture types.

Note that the APIC frames must be written at the beginning, so the muxer will buffer the audio frames until it gets all the pictures. It is therefore advised to provide the pictures as soon as possible to avoid excessive buffering.  Also keep picture small (ie. less than 1% of total mp3 size is a good rule/target to follow).

A Xing/LAME frame right after the ID3v2 header (if present). It is enabled by default, but will be written only if the output is seekable. The write_xing private option can be used to disable it. The frame contains various information that may be useful to the decoder, like the audio duration or encoder delay.

A legacy ID3v1 tag at the end of the file (disabled by default). It may be enabled with the write_id3v1 private option, but as its capabilities are very limited, its usage is not recommended.  This is important for old mp3 players that only understand ID3v1 tags.

Examples:

On the next part you must change the input, out.mp3 and cover.png to your own fit.

1) Write an mp3 with an ID3v2.3 header and an ID3v1 footer:

ffmpeg -i input.mp3 -id3v2_version 3 -write_id3v1 1 out.mp3

2) To attach a picture to an mp3 file select both the audio and the picture stream with map:

ffmpeg -i input.mp3 -i cover.png -c copy -map 0 -map 1 \
-metadata:s:v title="Album cover" -metadata:s:v comment="Cover (Front)" out.mp3

3) Write a "clean" MP3 without any extra features:

ffmpeg -i input.wav -write_xing 0 -id3v2_version 0 out.mp3

Let's get work done:
So this is what I did to add a cover image to a mp3 that already had id3tags:

ffmpeg -i orignal.mp3 -i cover.jpg -map 0:0 -map 1:0 -c copy -id3v2_version 3 -metadata:s:v comment="Cover (Front)" final.mp3

Other thoughts:

I did use a jpeg image for the cover and ffmpeg accepted it as well as vlc, mpv and other players I tested did work fine.

So if you need to include the same cover on many files (ie an album) in a folder, you can do this:

1) "cd" to the target folder
2) "mkdir new" in the target folder
3) rename the cover image file to cover.<ext> where <ext> could be png or jpg depends on your choice and particular case, or convert it to jpg.
3) then copy and past the following script (and hit the [ENTER] key):

for file in *.mp3; do ffmpeg -i "${file}" -i cover.jpg \
 -map 0:0 -map 1:0 -c copy -id3v2_version 3 -metadata:s:v \
 comment='Cover (Front)' ./new/"${file}"; done

After it runs you will have a sub-folder with the new mp3 files all with the embedded cover image you inserted with the script.

Have fun!  Cheers!